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EXECUTIVE  SUMMARY 


One  idea  of  the  free  flight  concept  suggests  shifting  aircraft  separation  responsibility  from  air 
traffic  controllers  to  flight  crews.  This  creates  a  ‘shared-separation’  authority  environment. 
Potential  benefits  of  shared-separation  in  free  flight  include  improved  safety  through  enhanced 
conflict  detection  and  resolution  capabilities,  more  flexibility  to  manage  flight  operations,  and 
better  decision-making  tools  for  air  traffic  controllers  and  flight  crews.  The  Federal  Aviation 
Administration  (FAA),  National  Aeronautics  and  Space  Administration  (NASA),  and  the  Volpe 
National  Transportation  Systems  Center  completed  the  first  integrated,  high  fidelity,  real-time, 
human-in-the-loop  simulation  study  of  this  concept  in  February  2000.  The  FAA  (AAR- 100, 
ASD-130,  and  ATP-400)  and  NASA  Ames  Research  Center  (ARC)  (Advanced  Air 
Transportation  Technologies  Program)  co-sponsored  the  study,  termed  Air-Ground  Integration 
Experiment  (AGIE). 

AGIE  provided  an  initial  examination  of  the  effect  of  shared-separation  authority  on  flight 
operations  when  both  air  and  ground  have  enhanced  traffic  and  conflict  alerting  systems.  The 
NASA  ARC  developed  the  Cockpit  Display  of  Traffic  Information  with  Alerting  Logic 
(CDTI-AL)  prototype,  which  served  as  the  decision  support  tool  for  the  flight  crews.  The 
MITRE  Corporation-developed  User  Request  Evaluation  Tool  (URET)  was  available  to  air 
traffic  controllers.  The  objectives  of  the  study  were:  to  identify  operational  issues  (e.g., 
communications,  procedures)  that  affect  shared-separation  operations,  to  provide 
reconunendations  for  the  information  requirements  and  procedures  necessary  to  facilitate  shared- 
separation  operations,  and  to  evaluate  the  effect  of  shifting  separation  authority  on  controller  and 
pilot  workload  and  situation  awareness. 

AGIE  was  conducted  concurrently  using  simulation  facilities  located  at  the  FAA  William  J. 
Hughes  Technical  Center  on  the  east  coast  and  NASA  ARC  on  the  west  coast.  The  simulation, 
conducted  over  a  4- week  period,  included  six  pilot  participants,  12  certified  professional 
controllers,  and  four  operations  supervisors  as  study  participants.  Expert  observers  (EO),  who 
were  subject  matter  experts,  observed  the  simulation  and  recorded  interesting  observations. 

Two  Memphis  Air  Route  Traffic  Control  Center  (ARTCC)  sectors,  sectors  21  and  44,  were 
emulated  in  the  experiment.  All  adjacent  surrounding  sectors  were  combined  on  a  single 
position,  collectively  called  sector  78,  which  was  staffed  by  a  member  of  the  experiment  team. 
The  simulation  consisted  of  four  conditions  defined  by  various  levels  of  controller  and  flight 
crew  shared-separation  responsibilities.  The  conditions  were  Current  Operations  (CO),  CO  with 
CDTI-AL  (CO:CDTI),  Shared-Separation  Level  1  (SS:L1),  and  Shared-Separation  Level  2 
(SS:L2).  Each  condition  used  a  different  set  of  procedures  that  reflected  changing  roles  and 
responsibilities  for  the  participants.  Current  standard  separation  rules  of  5  nm  horizontal  or 
1000/2000  ft  vertical  (as  appropriate)  were  observed  for  all  conditions.  All  flight  crew  and 
controller  participants  were  exposed  to  each  condition,  creating  a  within-subjects  design. 

Scenarios  were  developed  from  flight  plans  extracted  from  Memphis  ARTCC  System  Analysis 
and  Recording  tapes  and  accompanying  Adaptation  Control  Environment  System  configuration 
tapes  obtained  from  the  field.  TTie  data  allowed  for  the  realistic  representation  of  sector 


boundaries,  jet  routes,  and  fixes  for  the  simulated  sectors.  Each  of  the  four  data  collection  runs 
(CO,  CO:CDTI,  SS:L1,  and  SS:L2)  had  16  planned  conflicts,  8  in  each  sector,  involving  two 
aircraft  converging  at  acute  angles. 

Subjective  and  objective  data  were  collected  firom  participant  controllers  and  pilots,  the  air  traffic 
control  (ATC)  environment,  and  the  flight  deck.  TTie  ground-side  (controller  and  ATC) 
objective  data  included  communications,  separation  errors,  URET  alerts  and  trial  plans, 
minimum  separation  distance  (MSD),  traffic  density,  the  number  of  free  flight  cancellations,  and 
other  data.  The  air-side  (pilots  and  flight  deck)  objective  data  consisted  of  communications, 
separation  errors,  CDTI-AL  alerts,  MSD,  the  number  of  fi^  flight  cancellations,  and  other  data. 
Both  the  ground-side  and  air-side  subjective  data  consisted  of  workload,  situation  awareness 
ratings,  experiences  with  shared-separation,  traffic  realism,  and  other  details  using  post-run  and 
exit  forms.  EOs  also  recorded  some  critical  observations  such  as  free  flight  cancellations. 

In  general,  the  participant  controllers  had  concerns  regarding  the  feasibility  of  shared-separation 
conditions  as  simulated  in  this  study.  Controllers  reported  higher  workload  and  expressed  safety 
concerns  under  shared-separation  conditions,  which  was  demonstrated  by  their  free  flight 
cancellations.  Controllers  also  preferred  to  resolve  conflicts  earlier  than  pilots  and  tended  to 
cancel  free  flight  when  they  perceived  pilots  were  delaying  the  conflict  resolution.  However, 
their  level  of  situation  awareness  was  high  across  all  conditions.  The  pilot  participants  preferred 
shared-separation  conditions,  particularly  the  condition  in  which  they  had  the  highest  level  of 
separation  responsibility  (SS:L2).  They  rated  both  shared-separation  conditions  as  being 
relatively  safer  than  current  operations  and  as  providing  more  situation  awareness.  Although  it  is 
premature  to  identify  the  best  possible  shared-separation  level,  the  results  of  this  study 
demonstrate  the  need  to  conduct  further  research  in  this  area. 
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1.  INTRODUCTION 


One  element  of  the  free  flight  concept,  as  described  by  the  RTCA  Task  Force  3  (1995),  suggests 
placing  more  responsibility  on  flight  crews  to  maintain  safe  separation  from  other  aircraft  in  the 
National  Airspace  System  (NAS).  This  idea  could  potentially  shift  aircraft  separation 
responsibility  from  air  traffic  controllers  to  flight  crews  creating  a  '’shared-separation'  authority 
environment.  The  guiding  principle  of  the  free  flight  concept  is  to  provide  benefits  to  users  and 
service  providers.  Some  of  the  possible  benefits  include  improved  safety  through  enhanced 
conflict  detection  and  resolution  capabilities,  more  flexibility  to  manage  flight  operations,  and 
better  decision-making  tools  for  air  traffic  controllers  and  flight  crews.  To  exercise  these 
benefits,  there  may  be  a  need  to  supply  traffic  information  to  flight  crews,  and  develop  operating 
methods  and  tools  for  both  the  air  and  ground  to  assure  safety.  The  Air  Traffic  Services  Concept 
of  Operations  2005  (FAA,  1998)  promotes  similar  free  flight  ideas  for  shared-separation 
responsibility  that  also  include  trajectory  negotiation  between  the  users  and  air  traffic  controllers, 
user  collaboration  with  controllers  to  determine  optimal  schedules  and  trajectories,  and  training 
and  procedures.  In  addition,  recent  work  on  Distributed  Air-Ground  Traffic  Management  at  the 
National  Aeronautics  and  Space  Administration  (NASA)  (NASA,  1999)  reflects  some  of  these 
same  considerations  for  this  new  concept  of  operations. 

To  investigate  some  of  these  concepts,  the  NASA  Ames  Research  Center  (ARC)  has  developed  a 
Cockpit  Display  of  Traffic  Information  (CDTI)  prototype.  The  CDTI  includes  embedded 
conflict-alerting  logic  that  predicts  the  probability  of  an  encounter  with  another  aircraft.  The 
CDTI  with  alerting  logic  (CDTI-AL)  assumes  Automatic  Dependent  Surveillance-Broadcast 
(ADS-B)  technology  to  supply  the  position  and  trajectory  information  of  all  proximal  air  traffic. 
This  prototype  ‘decision  support  tool’  is  intended  to  enhance  flight  crew  situation  awareness  and 
provide  more  autonomy  in  the  NAS.  In  addition,  a  ground-based  conflict  probe  and  trial¬ 
planning  tool  has  been  developed  for  use  by  air  traffic  controllers.  This  prototype  decision 
support  tool,  entitled  User  Request  and  Evduation  Tool  (URET),  is  currently  fielded  for  daily 
use  at  the  Indianapolis  and  Memphis  Air  Route  Traffic  Control  Centers  (ARTCCs)  and  is  a  key 
component  of  the  Federal  Aviation  Administration  (FAA)  Free  Flight  Phase  I  Program.  There 
have  been  studies  done  on  each  of  these  tools  individually,  but  there  is  a  need  to  investigate  how 
they  might  impact  procedures  and  human  performance  in  a  shared-separation  environment. 

The  FAA,  NASA,  and  the  Volpe  National  Transportation  Systems  Center  (VNTSC)  have  begun 
a  collaborative  research  effort  to  explore  some  of  these  free  flight  issues.  The  first  integrated, 
high  fidelity,  real-time,  human-in-the-loop  simulation  of  a  planned  series  of  studies  began  in  Fall 
1999  and  was  completed  in  February  2000.  The  concept  exploration  study  termed  the  Air- 
Ground  Integration  Experiment  (AGIE)  was  co-sponsored  by  the  FAA  (AAR- 100,  ASD-130, 
ATP-400)  and  the  NASA  Advanced  Air  Transportation  Technologies  Project. 

1.1  Objectives 

This  experiment  provided  an  initial  examination  of  the  effect  of  shared-separation  authority  on 
flight  operations  when  both  air  and  ground  operators  have  enhanced  traffic  and  conflict  alerting 
systems.  There  was  a  strong  emphasis  on  identifying  and  evaluating  human  factors  issues.  The 
specific  objectives  were 
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•  to  identify  operational  issues  (e.g.,  communications  and  procedures)  that  affect  shared- 
separation  operations, 

•  to  provide  recommendations  for  the  information  requirements  and  procedures  necessary  to 
facilitate  shared-separation  operations,  and 

•  to  evaluate  the  effect  of  shifting  separation  authority  on  controller  and  pilot  workload  and 
situation  awareness. 

1.2  Literature  Review 


Several  studies  have  investigated  various  aspects  of  the  implementation  of  free  flight.  Many  of 
these  studies  have  focused  on  the  tools  that  pilots  and  controllers  will  require  to  allow  for 
increased  flexibility  in  routing  and  separation  responsibility.  For  example,  Pekela  and  Hilbum 
(1998)  conducted  a  free  flight  study  in  which  military  controllers  were  given  a  plan  view  display 
(PVD)  and  a  prototype  CDTI.  Although  the  CDTI  would  typically  reside  on  the  flight  deck,  this 
study  provided  a  CDTI  view  to  the  controller  for  experimental  purposes.  The  focus  of  the 
experiment  was  to  examine  workload,  visual  scanning,  and  monitoring  performance.  The  CDTI 
displayed  two  views  on  a  split  screen,  one  horizontal  (plan  view),  and  one  vertical  (altitude 
elevation  view).  The  CDTI  display  incorporated  conflict  detection  and  resolution,  which 
provided  a  view  similar  to  that  of  the  cockpit  view.  All  of  the  controllers  strongly  agreed  that  the 
CDTI  was  a  useful  tool.  Controllers  tended  to  rely  more  on  the  vertical  view  compared  to  the 
horizontal.  However,  under  high  traffic  conditions,  the  controllers  tended  to  revert  to  the  PVD. 
The  controllers  relied  more  on  the  CDTI  during  periods  of  conflict  and  more  on  the  PVD  during 
normal  operations.  Some  of  the  controllers  increased  reliance  on  the  CDTI  as  they  became  more 
familiar  with  its  operation.  In  terms  of  conflict  detection  performance,  there  were  no  clear 
advantages  between  the  PVD  and  CDTI  displays.  In  terms  of  resolution  detection  time,  there 
was  a  slight  advantage  to  the  use  of  the  CDTI  over  the  PVD.  Pekela  and  Hilbum  suggested  that 
there  is  a  need  to  redesign  air  traffic  control  (ATC)  displays  in  order  to  acconunodate  additional 
dimensional  approaches  to  conflict  resolution,  which  may  occur  more  frequently  in  flee  flight 
situations.  They  also  suggested  a  look-ahead  time  of  a  little  more  than  5  minutes  for  conflict 
probes. 

Kerns  (1999)  conducted  a  study  on  the  usefulness  of  URET  in  helping  controllers  manage  traffic 
in  an  unstructured  environment.  Controllers  judged  URET  to  have  a  favorable  impact  on  safety 
and  ATC  performance,  and  these  benefits  were  judged  most  pronounced  in  the  free  flight 
condition.  Endsley,  Sollenberger,  Nakata,  and  Stein  (2000)  also  reported  enhanced  displays  may 
provide  help  for  controllers  under  free  flight  conditions.  In  this  study,  the  simulated  enhanced 
display  contained  text  information  about  transitioning  aircraft  on  an  ATC  radar  display.  The 
researchers  recommended  further  exploration  of  the  concept  of  enhanced  displays  with  an  effort 
to  integrate  the  additional  display  information  with  the  controller  radar  picture. 

Other  studies  have  focused  on  the  effects  that  shared-separation  responsibility  may  have  on 
controller  workload,  situation  awareness,  and  perceived  safety.  Endsley,  Mogford,  Allendoerfer, 
Snyder,  and  Stein  (1997)  and  Endsley  (1997)  reported  that  controllers  acting  as  passive  monitors 
during  free  flight  may  show  a  decrease  in  situation  awareness,  might  show  an  increase  in 
workload  due  to  different  responsibilities,  and  have  problems  making  timely  interventions.  Their 
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work  also  indicated  that  communications  requirements  may  significantly  increase  under  free 
flight  conditions  due  to  the  need  for  controllers  to  accurately  obtain  pilot  intent  information,  and 
to  provide  additional  information. 

Hilbum,  Bakker,  and  Pekela  (1998)  reported  the  importance  of  giving  controllers  information  on 
aircrafr  intent  during  free  flight  situations.  There  were  no  more  errors  in  determining  possible 
separation  violations  between  the  conditions  of  intent  notification  and  without  intent  notification, 
but  controllers  reported  more  possible  conflicts  under  free  flight  with  no  intent,  compared  to  free 
flight  with  intent  scenarios.  Some  controllers  felt  that  sharing  intent  information  would  increase 
safety.  Hilbum  et  al.  also  reported  that  controller  subjective  and  objective  workload  could  be 
reduced  using  free  flight  compared  to  conventional  controlled  flight.  Under  low-density  traffic 
conditions,  workload  was  reduced  more  for  free  flight  with  intent  scenarios  compared  to  free 
flight  without  intent  scenarios.  Under  high  traffic  conditions,  there  was  no  apparent  reduction  in 
controller  workload  because  of  shared  intent  information.  In  addition,  several  controllers  in  this 
study  expressed  concern  with  reliance  on  automated  conflict  detection  tools. 

In  a  study  conducted  by  Corker,  Fleming,  and  Lane  (2001),  controllers  managed  traffic  under 
varying  levels  of  separation  authority.  Different  mixes  of  free  flight  equipage  were  manipulated 
to  examine  the  potential  effects  of  equipage  upon  distribution  of  separation  authority.  Their 
findings  indicated  that  controllers  were  more  likely  to  take  direct  control  of  aircrafr  in  shared- 
separation  scenarios  with  the  addition  of  more  aircrafr  maintaining  their  own  separation.  The 
study  also  revealed  that  when  a  majority  of  the  aircrafr  were  managing  their  own  separation,  the 
subjective  workload  ratings  for  the  controllers  were  reported  as  high.  That  increase  in  workload 
appeared  to  be  directly  related  to  the  increase  in  communication  requirements  necessary  to 
accomplish  the  controllers’  management  of  their  airspace.  Thus,  the  investigation  emphasized 
the  importance  of  providing  tactical  intent  information  to  the  controllers  in  free  flight  operations 

There  have  also  been  some  studies  exploring  the  effects  of  free  flight  operations  on  flight  crews. 
Collaborative  studies  conducted  by  NASA  and  the  Netherlands  National  Aerospace  Laboratory 
(Mackintosh  et  al.,  1998)  examined  flight  crew  procedures  in  free  flight  operations.  Each  of  the 
investigations  examined  the  effects  of  traffic  density  on  the  flight  crew  participants.  Crews  were 
provided  with  prototypic  airborne  alerting  logic  and  CDTI  display  tools  to  help  enable  the  flight 
crew  separation  tasks.  Both  studies  found  longer  conflict  detection  times  in  high  density 
compared  to  low-density  traffic  scenarios.  The  NASA  investigation  also  included  controllers  as 
participants.  In  that  investigation,  both  flight  crews  and  controllers  appeared  to  have  some 
performance  differences  based  on  the  different  geometry  of  the  conflict  angles. 

Some  flight  deck  research  has  also  begun  to  explore  the  usefulness  of  free  flight  tools.  Johnson, 
Battiste,  and  Bochow  (1999)  have  provided  some  guidelines  related  to  CDTI  features  that  might 
be  required  in  a  free  flight  operational  environment.  Their  research  suggests  the  importance  of 
color  coding  and  3-D  flight  plans  for  alerting  and  situation  awareness.  These  features  and  their 
display  characteristics  may  be  critical  to  the  successful  implementation  of  shared-separation. 

Another  aspect  of  free  flight  that  has  been  discussed  in  the  literature  is  the  characteristics  of  the 
airborne  alerting  logic.  As  a  tool,  this  logic  will  assist  the  flight  crews  in  detecting  and  resolving 
conflicts  in  a  shared-separation  environment.  Cashion  and  Lozito  (2000)  examined  the  impact  of 
different  levels  of  intent  in  the  airborne  alerting  logic  on  flight  crews.  They  found  that  crews 
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prefer  longer-term  intent  [i.e.,  aircraft  intent  that  includes  horizontal  and  vertical  navigation 
components  of  the  Flight  Management  System  (FMS)].  However,  the  flight  crews  expressed 
concern  about  display  clutter  when  portraying  more  intent  data  on  the  CDTI. 

Smith,  Billings,  McCoy,  and  Orasanu  (1999)  reviewed  other  free  flight  issues.  Their  findings 
suggested  there  may  be  advantages  to  allowing  pilots  to  have  additional  tools  available  such  as 
enhanced  weather  displays,  conflict  alert  probes,  and  others.  The  research  suggested  that  the 
decision-making  process  becomes  increasingly  complex  as  communications  increase  and  more 
decision  makers  are  placed  into  the  loop.  Controllers  may  become  less  efficient  and  less  able  to 
retain  awareness  of  traffic  situations  if  their  management  role  is  changed  to  the  position  of  a 
monitor  of  a  highly  complex  automated  system.  The  research  also  showed  that  a  key  to 
improving  operations  may  involve  cooperative  flight  planning  and  the  sharing  of  information 
concerning  routine  bottlenecks  or  constraints. 

In  summary,  previous  research  showed  the  need  to  develop  both  tools  such  as  CDTI  and  ground 
conflict  probes  and  procedures  to  deal  with  issues  unique  to  fi'ee  flight.  The  research  suggested 
that  roles  and  responsibilities  of  both  the  pilots  and  controllers  need  to  be  clearly  defined.  The 
following  is  a  list  of  free  flight  issues  identified  from  those  studies  that  may  require  further 
investigation: 

•  Impact  on  controller  and  pilot  workload  and  situation  awareness. 

•  Impact  on  communications  due  to  exchange  of  information  and  inquiry. 

•  Ability  of  controllers  to  make  timely  interventions  to  resolve  conflicts. 

•  Need  for  pilot  intent  information. 

•  Need  for  additional  automation  to  assist  with  the  management  of  information. 

•  Assessment  and  development  of  procedures.  Development  of  decision  support  tools  and 
displays. 

•  Impact  of  aircraft  mix  on  operations. 

•  Integrated  evaluation  of  the  above. 

This  study  was  designed  to  address  and  investigate  aspects  of  these  issues  (with  the  exception  of 
the  last  one). 

2.  METHOD 


Typically,  fast-time  simulation,  modeling,  paper  studies,  part-task  and  lower  fidelity  real-time 
human-in-the-loop  simulation  studies  provide  preliminary  assessments  of  advanced  concepts 
such  as  shared-separation.  The  literature  review  indicated  that  a  number  of  such  studies  were 
done.  However,  the  literature  identified  a  scarcity  of  data  from  an  air-ground  integrated 
perspective.  Although  shared-separation  concepts  are  not  matured,  the  researchers  felt  that 
conducting  a  high  fidelity  simulation  would  identify  a  direction  for  ftuther  research  and  examine 
early  feasibility  and  benefits  from  an  integrated  perspective.  Therefore,  researchers  chose  a  high 
fidelity  infi^tructure.  Additionally,  by  conducting  simulation  in  high  fidelity  laboratories,  the 
researchers  attempted  to  eliminate  the  effect  of  nuisance  variables. 
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2.1  Participants 

Participants  included  air  traffic  controllers  and  line  pilots.  Participants  were  organized  in  groups 
consisting  of  four  controllers  (a  radar  [R-side]  controller  and  a  radar  associate  [D-side]  controller 
team  per  sector)  and  two  line  pilots  (a  flight  crew). 

2.1.1  Controllers 

Three  groups  of  Certified  Professional  Controllers  (CPCs)  and  one  group  of  Operations 
Supervisors  (OSs)‘  from  Memphis  ARTCC  (ZME)  participated  in  the  simulation  as  air  traffic 
controllers.  Each  group  consisted  of  two,  2-member  teams  and  participated  for  3  days  during  the 
4  weeks  of  simulation.  Each  sector  was  staffed  with  an  R-side  controller  and  a  D-side  controller. 
All  CPCs  and  supervisors  were  qualified  to  control  traffic  in  the  sector  and  position  they  were 
assigned  to  operate.  Their  sector  and  position  assignment  did  not  change  throughout  the 
simulation.  Table  1  summarizes  the  participant  demographic  information. 


Table  1.  Summary  of  Background  Form  Responses 


Characteristics 

Certified  Professional 
Controllers  (n=12) 

Operations  Supervisors 
(n=4) 

M  SD 

M  SD 

Age 

37.9  years 

3.2  years 

43.5  years 

4.2  years 

Total  experience  as 
Developmental-CPC  /  CPC 

14.9  years 

3.7  years 

19.6  years 

4.5  years 

Experience  as  an  FAA 
Developmental-CPC  /  CPC 

14.1  years 

3.1  years 

17.8  years 

4.1  years 

Experience  as  a  CPC 

1 1 .9  years 

3.4  years 

15.9  years 

5.1  years 

Years  at  ZME 

12.6  years 

3.1  years 

15,5  years 

1.9  years 

Years  of  URET  usage 

1 .4  years 

0  years 

URET  usage  at  ZME  sector  21 

92.5% 

9.9% 

100% 

0% 

URET  usage  at  ZME  sector  44 

93.3% 

12.1% 

100% 

0% 

Overall  URET  usage 

86.7% 

12.6% 

N/A 

N/A 

'  Unfortunately  CPCs  were  not  available  from  the  field  for  the  last  week  of  the  study,  therefore  OSs  acted  as 
participants  for  one  week.  Data  from  the  OSs  were  not  included  in  data  analyses  except  for  background  statistics 
and  comments  from  their  forms  and  debriefing  sessions. 
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2.1.2  Pilots 


Three  flight  crews,  consisting  of  both  captains  and  first  officers  from  a  major  United  States 
airline  served  as  participants.  All  pilots  were  either  current  on  the  Boeing  747-400  or  retired  for 
not  more  than  6  months.  Pilots  flying  the  Boeing  747-400  typically  fly  oceanic  routes;  therefore, 
all  participants  in  this  study  were  oceanic  line  pilots  to  avoid  training  concerns.  The  pilots  flew 
in  their  normal  crew  position.  The  captains  had  a  mean  total  flight  time  of  18,000  hours,  and  the 
first  officers  had  a  mean  total  flight  time  of  16,930  hours. 

2.2  Simulation  Experiment  Team 

The  simulation  experiment  team  consisted  of  two  test  directors  (one  at  the  William  J.  Hughes 
Technical  Center  [WJHTC]  and  one  at  NASA  ARC),  human  factors  researchers,  subject  matter 
experts,  statisticians,  laboratory  personnel,  and  audio/video  personnel  from  the  FAA,  NASA,  and 
VNTSC. 

The  test  directors  were  responsible  for  the  overall  management  of  the  simulation  and  directed 
simulation-related  activities  of  all  members  of  the  simulation  experiment  team. 

Human  factors  researchers,  subject  matter  experts,  statisticians,  and  trained  simulation  pilots 
staffed  the  experiment  team  positions.  Experiment  team  members  also  administered  forms  and 
conducted  participant  briefings  and  debriefings.  Laboratory  persormel  operated,  monitored,  and 
maintained  the  laboratory  systems  used  in  the  simulation.  All  team  members  were  available  in 
the  test  areas  to  support  the  test  directors. 

2.2.1  Expert  Observers 

Two  expert  observers  (EOs)  participated  as  part  of  the  simulation  experiment  team  during  each 
of  the  4  study  weeks.  One  EO  observed  Sector  21  and  the  other  observed  Sector  44.  EOs  were 
subject  matter  experts  in  the  field  of  ATC. 

2.2.2  Ghost  Sector  Controller 

Two  members  of  the  simulation  experiment  team  staffed  the  “ghost  sector”  controller  position 
for  all  adjacent,  non-simulated  sectors.  The  ghost  controllers  accepted  and  made  hand-offs  and 
performed  airoground  and  groundoground  (land  line)  communications  as  required.  These 
individuals  were  trained  on  the  necessary  equipment,  airspace,  and  sector  operating  procedures. 

2.2.3  Automatic  Datalink  Operator 

One  member  of  the  simulation  experiment  team  staffed  the  automatic  datalink  operator  (ADO) 
position  during  specific  runs  as  appropriate.  Two  individuals  were  trained  for  the  ADO  position 
and  alternated  throughout  the  study.  The  ADO  had  prior  knowledge  of  all  scripted  altitude  and 
course  changes  and  monitored  the  air<^air  and  test  director  frequencies  for  any  unexpected/ 
unknown  altitude  or  course  changes.  The  ADO,  situated  at  a  separate  Display  System 
Replacement  (DSR)  console,  updated  the  Host  computer  vwth  the  required  changes.  This 
ensured  that  the  Host,  CDTl-AL,  and  URET  remained  current  and  consistent  with  flight  plan 
updates. 
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2.2.4  WJHTC  Simulation  Pilots  and  Laboratory  Coordinators 


Ten  trained  WJHTC  simulation  pilots  and  two  laboratory  coordinators  from  the  simulation 
experiment  team  supported  the  two-sector  operation.  Simulation  pilots  emulated  pilot 
communications  and  actions.  They  initiated  scripted  air<->air  and  air->ground  communications 
and  responded  to  ATC  instructions.  The  simulation  pilots  also  entered  data  into  the  desktop 
simulators  as  required  by  the  scripts  and  in  response  to  controller-issued  instructions  (e.g.,  turn 
right  heading  120,  climb  to  and  maintain  Flight  Level  (FL)  270,  etc).  Eighty  percent  of  the 
simulation  pilots  were  licensed  or  retired  pilots. 

2.2.5  Intruder  Simulation  Pilot 

One  trained  simulation  pilot  at  NASA  ARC  staffed  the  intruder  aircraft  simulator  that  was 
scripted  to  be  involved  in  planned  conflicts^.  The  intruder  simulation  pilot  was  trained  on  the 
use  of  CDTI-AL  and  the  right-of-way  rules.  This  individual  also  had  access  to  communications 
with  both  the  controllers  and  flight  crew  participants. 

2.3  Facilities  and  Equipment  Overview 

The  simulation  test  bed  integrated  facilities  and  equipment  from  both  the  WJHTC  and  NASA 
ARC.  WJHTC  facilities  included:  Integration  and  Interoperability  Facility  (I^F),  Target 
Generation  Facility  (TGF),  and  the  Pseudo  Aircraft  Systems  (PAS)  laboratory.  The  I  F 
equipment  configuration  included:  Host  processor,  DSR  consoles,  URET,  voice  communication 
system,  audio  and  video  recording  system,  and  workload  assessment  keypads  (WAKs).  The 
NASA  ARC  facility  used  was  the  Crew-Vehicle  Systems  Research  Facility  (CVSRF).  The 
CVSRF  laboratory  and  equipment  configuration  included:  NASA  ARC  Boeing  747-400  flight 
simulator  (NASA  ARC  simulator),  alerting  logic,  flight  crew  displays  and  tools,  PAS  laboratory, 
intruder  aircraft  simulator,  voice  communication  system,  and  audio  and  video  recording  system. 

The  WJHTC  and  NASA  ARC  laboratories  were  linked  across  the  country  via  a  high-speed 
circuit  (fractional  T1  line)  that  digitally  transmitted  data  and  voice.  Figure  1  depicts  the 
integration  of  the  facilities. 


^  A  planned  conflict  was  defined  as  two  aircraft  on  flight  paths  that  would  collide  if  there  was  no  corrective  action 
taken. 
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Figure  1 .  FAA  and  NASA  facilities  used  in  AGIE. 
2.3.1  WJHTC  Facilities  and  Equipment 


The  I^F  is  devoted  to  exploring  the  issues  associated  with  modernizing  the  NAS  infrastructure. 
The  I^F  provides  a  realistic  DSR  en  route  environment  for  research.  It  is  open  to  modifications 
of  both  hardware  and  software  to  facilitate  the  conduct  of  engineering  evaluations.  The  I  F  is 
designed  for  prototype  experimentation,  system-level  integration,  proof  of  concept  evaluations 
(i.e.  shared-separation  operations),  and  interoperability  verification  and  evaluations.  The  I  F 
sector  configuration  for  this  experiment  is  depicted  in  Figure  2. 
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Figure  2.  I^F  controller  workstation  and  sector  layout. 

•  Display  System  Replacement 

DSR  consoles  replaced  existing  en  route  display  systems  with  new  hardware  and  software  at 
the  ARTCC.  DSR  provided  Common  Console  Workstations  on  high-resolution  20  inch  by 
20  inch  color  display  screens.  DSR  utilizes  Reduced  Instruction  Set  Computer  processing 
technology  and  serves  as  a  platform  for  future  ATC  system  upgrades.  The  DSR  version  used 
in  this  experiment  was  BAB03. 

•  User  Request  Evaluation  Tool 

URET  was  developed  to  assist  the  ARTCC  controller  in  predicting  and  evaluating  potential 
conflicts  between  aircraft.  URET  is  currently  installed  as  a  prototype  system  and  fielded  for 
daily  use  at  the  Memphis  and  Indianapolis  ARTCCs.  The  system  fimctionality  consists  of 
trajectory  modeling,  conformance  monitoring  and  reconformance,  current  plan  and  trial  plan 
processing,  automated  problem  detection,  interfaces  with  the  Host  and  external  data  sources, 
and  computer-human  interface. 

URET  provides  the  controller  five  levels  of  automated  problem  detection  alerts  with  a  "look¬ 
ahead"  time  of  approximately  20  minutes.  The  following  alerts  are  presented  to  the 
controller  in  both  tabular  and  graphic  form: 
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-  RED  Alert  -  The  alert  given  if  aircraft  are  predicted  to  pass  within  the  standard 
separation  limits  of  five  nm  horizontally  and  1000/2000  ft  vertically  (as  appropriate^) 

-  Muted  RED  Alert  -  The  alert  given  if  the  predicted  separation  between  two  aircraft  is 
less  than  the  standard  separation  limits,  and  the  separation  loss  is  predicted  to  occur  on  a 
portion  of  the  route  where  an  altitude  transition  is  planned,  but  not  yet  cleared. 

-  YELLOW  Alert  -  The  alert  given  if  aircraft  are  predicted  to  pass  between  5  to  10  miles 
horizontally  and  within  1000/2000  ft  vertically  (as  appropriate). 

-  Muted  YELLOW  Alert  -  The  alert  given  if  the  predicated  separation  between  two  aircraft 
is  between  5  to  10  miles  horizontal,  less  than  1000/2000  ft  vertical,  and  the  separation 
loss  is  predicted  to  occur  on  a  portion  of  the  route  where  an  altitude  transition  is  planned, 
but  not  yet  cleared. 

-  BLUE  Alert  -  The  alert  given  if  aircraft  are  predicted  to  enter  Special  Use  Airspace 
(SUA)  based  on  their  current  trajectory. 

The  URET  version  used  for  this  study  was  D32.  This  version  provided  a  two-way  Host 
interface  allowing  controllers  to  amend  flight  plans  directly  through  URET.  To  mimic 
operations  in  the  field,  paper  flight  strips  were  not  provided  to  participants.  (URET 
information  was  used  in  lieu  of  the  flight  strips.) 

•  Voice  Communication  Systems 

The  PF  and  CVSRF  integrated  their  laboratory  voice  communication  and  recording  systems 
to  emulate  the  operational  ATC  and  flight  deck  systems  in  use  today. 

Voice  communications  between  the  facilities  were  transmitted  over  a  leased  digital  circuit 
(fractional  Tl).  This  circuit  carried  voice  information  in  Internet  Protocol  (IP)  packets.  One 
Cisco  3640  Voice  Over  IP  router  was  located  at  each  end  of  the  Tl  circuit  and  provided  an 
interface  between  the  individual  voice  conununications  systems  and  the  Tl. 

The  system  provided  six  separate  voice  frequency  channels  for  the  simulation.  These 
included  an  air^^air  channel,  air<->^ground  channels  for  each  of  the  two  sectors  simulated  and 
the  ghost  sector,  a  land  line  channel  for  all  sectors,  and  an  additional  channel  for 
communications  between  test  directors  at  NASA  and  WJHTC. 

Pilots  were  able  to  transmit  on  the  air^air  and  air<->ground  frequencies.  Pilots  could  neither 
monitor  nor  transmit  on  the  land  line  channels.  Controllers  were  able  to  transmit  on  the  land 
line  and  air<^ground  frequencies.  Controllers  were  able  to  monitor  but  not  transmit  on  the 
air<->air  channel.  The  controllers  and  pilots  used  their  headphones  to  access  all  frequencies; 
no  loud  speaker  or  other  equipment  was  provided.  Controllers  and  pilots  were  able  to 
simultaneously  monitor  both  the  air<^air  channel  and  their  specific  air<->ground  channels. 


^  Vertical  separation  requirements  for  aircraft  flying  below  flight  level  (FL)  290  is  1000  ft.  Vertical  separation 
requirements  for  aircraft  flying  above  FL290  is  2000  ft. 
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•  ATC  Audio  and  Video  Recording  System 

A  mobile  recording  system  was  used  to  record  the  audio  and  video  data  during  each 
simulation  run.'*  Four  black  and  white,  low-light  micro  cameras  recorded  two  views  of  each 
sector.  One  camera  was  focused  on  a  general  sector  overview,  and  the  second  camera  was 
focused  on  the  URET  display  for  each  sector.  All  videos  were  recorded  in  the  Super  VHS 
format  on  tapes  stamped  with  National  Television  System  Committee  linear  time  code  for 
synchronous  playback  purposes. 

Ambient  commimications  were  recorded  from  wireless  microphones  worn  by  each  controller. 
Land  line,  air^^ground,  and  air<-»air  voice  commimications  were  separately  recorded.  All 
audio  and  video  signals  were  mixed  using  a  Tascam  M25 16  audio  mixing  board  and  recorded 
on  the  hi-fi  audio  channels  of  the  videotapes. 

•  Workload  Assessment  Keypad 

A  WAK  was  provided  to  each  controller  position.  Using  the  WAK,  instantaneous  controller 
workload  ratings  on  a  l-to-5  scale  (1  =  very  low,  3  =  moderate,  and  5  =  very  high)  were 
collected  at  5-minute  intervals.  Four  WAK  units  were  connected  to  one  laptop.  This  laptop 
hosted  the  WAK  software  and  recorded  the  data  entered.  The  software  emitted  a  low  level 
beep  every  five-minutes  on  all  four  WAKs  simultaneously.  At  the  same  time,  the  keys  were 
illuminated  for  a  maximum  of  20  seconds.  If  a  participant  did  not  enter  a  workload  rating  in 
20  seconds,  the  WAK  automatically  recorded  an  entry  of  99  to  indicate  missing  data.  Figure 
3  depicts  the  WAK. 


Figure  3.  Workload  assessment  keypad. 

2.3. 1.2  Target  Generation  Facility 

The  TGF  generated  high  fidelity  digital  radar  messages  for  targets  in  the  simulated  airspace 
environment.  The  messages  were  adapted  to  mimic  actual  NAS  characteristics  by  including  the 
radar  and  environmental  characteristics  of  the  ZME.  Simulated  primary  and  beacon  radar  data 
were  generated  for  each  target  and  processed  by  the  multiple  radar  processing  function  of  the 


*  Video  and  audio  recordings  were  exclusively  used  to  backup  information  obtained  by  other  means.  They  also 
provided  a  mechanism  to  explore  issues  that  may  have  been  unclear  in  the  objective  and  subjective  data  obtained 
during  this  simulation.  The  information  contained  on  these  tapes  was  not  be  used  for  any  other  purpose.  All  t^)es, 
so  obtained,  are  held  by  ACT-540  and  were  made  available  only  to  the  members  of  the  experiment  team  and  to 
personnel  designated  by  NATCA.  All  tapes  were  destroyed  following  publication  of  the  final  report. 
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NAS  in  a  manner  similar  to  normal  radar  data.  Flight  data  blocks  contained  the  flight 
identification,  beacon  code,  and  altitude.  Target  positions  were  automatically  updated  at  the 
same  rate  that  is  experienced  in  the  ARTCC.  To  simulate  actual  aircraft  operations,  the  radar 
targets  maneuvered  based  on  route  segments  from  a  flight  plan  and  by  the  actions  of  the 
simulation  pilots  and  participant  pilots. 

2.3. 1.3  Pseudo  Aircraft  Systems  Laboratory 

PAS  is  a  computerized  flight  dynamics  and  piloting  system  designed  to  provide  a  high  fidelity, 
multi-aircraft,  and  real-time  simulation  environment  to  support  ATC  research.  PAS  is  comprised 
of  three  major  software  components  that  run  on  a  network  of  computer  workstations:  the 
Simulation  Manager,  the  PAS  Pilot  Manager,  and  the  Pilot  Station  Laboratory.  Combined,  the 
components  simulated  the  following  functions:  aircraft  performance  characteristics  and  flight 
dynamics,  flight  plans,  aircraft  state  information,  and  display  and  control  capabilities  for  the 
simulated  aircraft  on  a  set  of  workstations.  Both  NASA  and  the  WJHTC  used  PAS  laboratories 
with  the  same  version  of  PAS  software  (version  pas_4.3.2)  in  their  laboratories.  The  WJHTC 
PAS  laboratory  had  twelve  pilot  workstations  configured  for  AGIE.  Ten  workstations  were  for 
WJHTC  simulation  pilots  and  two  were  for  laboratory  coordinator  positions. 

2.3.2  NASA  Crew-Vehicle  Systems  Research  Facility  and  ARC  Simulator 

The  CVSRF  is  a  unique  national  research  facility  dedicated  to  studies  of  aviation  human  factors 
and  airspace  operations  and  their  impact  upon  aviation  safety.  An  integral  component  of  the 
CVSRF  is  the  NASA  ARC  simulator. 

The  NASA  ARC  simulator  was  built  by  CAE  Electronics  and  certified  to  the  FAA  Level  D 
certification  requirements  (Sullivan  &  Soukup,  1996).  The  Boeing  747-400  has  an  advanced 
level  of  automation  available  to  the  pilots.  The  visual  system  uses  photo  texturing  and  offers 
superior  scene  quality,  depicting  out  the  window  scenes  in  night,  day,  dusk,  or  dawn  conditions. 
In  addition,  the  simulator  has  an  advanced  digital  control  loading  and  a  six  degree-of-freedom 
motion  system.  Features  added  to  the  simulator  to  support  this  research  included  new  display 
components  (CDTI)  and  the  input  control  devices  for  the  display  (see  Figure  4).  Data  collection 
is  available  for  user  interaction  with  all  subsystems,  including  the  autopilot  system  and 
communication  devices. 
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Figure  4.  NASA  ARC  simulator  flight  deck  layout. 

2.3.2. 1  Alerting  Logic 

This  study  included  a  prototype  airborne  alerting  logic  designed  to  aid  in  shared-separation 
operations  (Yang  &  Kuchar,  1997).  This  alerting  logic  overlaid  the  NASA  ARC  simulator’s 
Traffic  Alert  and  Collision  (TCAS)  logic.  TCAS  involves  immediate  tactical  conflict  avoidance 
whereas  the  new  airborne  alerting  logic  was  designed  to  help  flight  crews  manage  the  more 
strategic  shared-separation  responsibilities.  The  goal  was  to  create  a  seamless  relationship 
between  the  airborne  alerting  logic  and  TCAS  (see  Figure  5). 
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Figure  5.  Relationship  of  new  airborne  alerting  logic  to  TCAS  logic. 

Therefore,  TCAS  was  left  intact  with  the  exception  that  the  first  two  threat  levels  of  display 
symbology  (unfilled  diamond  and  filled  diamond)  were  replaced  with  the  experimental  display 
symbology.  The  yellow  circle  for  a  Traffic  Advisory  (TA)  and  a  red  square  for  a  Resolution 
Advisory  (RA)  were  still  available.  Currently,  the  TCAS  display  depicts  surrounding  traffic  up 
to  40  nm  from  the  NASA  ARC  simulator  on  the  navigation  display.  In  contrast,  the  alerting 
logic  in  this  study  extended  traffic  depiction  out  to  120  nm  in  front  of  and  to  each  side  of  the 
NASA  ARC  simulator  and  30  nm  behind  the  NASA  ARC  simulator  based  on  the  expected 
ADS-B  surveillance  capabilities  (RTCA,  1992).  Additionally,  based  on  expected  ADS-B 
capabilities,  the  update  rate  for  the  navigation  display  was  once  per  second.  To  reduce  clutter,  an 
altitude  filter  limited  the  vertical  range  of  viewable  traffic  to  4100  ft  above  and  below  the  NASA 
ARC  simulator. 

The  airborne  alerting  logic  provided  an  additional  alerting  zone  beyond  that  of  TCAS.  The  alert 
was  provided  to  the  flight  crews.  A  CDTI-AL  alert  was  triggered  for  the  flight  crews  when  the 
alerting  logic  predicted  a  pending  violation  of  the  protected  zones  (or  minimum  separation 
requirement)  of  the  aircraft  (Yang  &,  Kuchar,  1997).  Operationally,  the  CDTI-AL  alert  is  the 
point  at  which  intervention  may  be  required  (RTCA,  1995). 

2.3.2.2  Flight  Crew  Displays  and  Tools 

Traffic  was  represented  on  the  flight  deck  navigation  display  by  the  symbol  “V”  with  the  apex 
indicating  the  aircraft  direction.  Altitude  was  pilot  selectable  as  altitude  relative  to  the  NASA 
ARC  simulator  or  absolute  altitude.  All  traffic  was  initialized  as  non-threat  aircraft.  In  addition, 
all  new  display  features  for  non-conflicting  aircraft  were  in  white.  Figure  6  depicts  all  aircraft  in 
a  non-threat  status. 
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Figure  6.  CDTI-AL  depicting  non-threat  aircraft. 

When  the  probability  of  a  violation  of  the  protected  zone  increased,  a  CDTI-AL  alert  was 
indicated  to  the  flight  crew  by 

•  a  blue  line  extending  from  both  the  NASA  ARC  simulator  and  the  intruder  aircraft  symbols. 
At  the  end  of  each  line  was  a  blue  circle  that  represented  the  current  separation  standard  of  5 
nm  in  diameter.  Any  overlap  of  the  circles  indicated  impending  loss  of  horizontal  separation. 

•  an  aural  warning  "alert"  sounded  twice; 

•  the  word  "ALERT"  appeared  in  blue  on  the  lower  right  hand  comer  of  the  display  along  with 
the  intruder  aircraft  call  sign  and  the  time  to  minimum  separation  distance  (MSD). 

The  time  to  MSD  was  the  time  remaining  before  aircraft  were  projected  to  pass  in  closest 
proximity  to  each  other  on  current  flight  paths.  All  display  features  associated  with  the  aircraft 
involved  in  a  CDTI-AL  alert  (aircraft  symbol,  altitude,  ground  speed,  and  callsigns)  as  well  as 
the  display  changes  related  to  an  alert  appeared  in  blue  to  help  identify  which  aircraft  were 
predicted  to  conflict  Figure  7  illustrates  the  display  changes  associated  with  a  CDTI-AL  alert. 
As  flight  crews  solved  a  conflict,  the  alert  level  degraded  to  a  non-threat  status  as  the  threat 
probability  was  reduced. 

Flight  crews  also  could  select  certain  display  features  designed  to  aid  them  in  shared-separation 
responsibilities.  Selectable  display  features  could  be  manipulated  by  a  small  box  mounted  above 
the  Mode  Control  Panel  (see  Figure  8).  Flight  crews  could  reduce  clutter  by  toggling  a  button  to 
de-select  the  traffic  callsigns.  Another  selectable  feature  was  the  temporal  predictor.  The 
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Figure  7.  CDTI-AL  depicting  an  alert. 


Figure  8.  Control  box  for  pilot  selectable  features. 

predictor  provided  crews  with  an  estimation,  based  on  current  aircraft  state  information,  of  where 
other  aircraft  would  be  relative  to  the  NASA  ARC  simulator  up  to  10  minutes  into  the  future. 

The  selection  knob  for  the  temporal  predictors  allowed  crews  continuous  control  of  the  predictor 
length  from  0  to  10  minutes  at  1  second  intervals.  Although  predictor  manipulation  does  not 
invoke  the  alerting  logic,  crews  could  visually  determine  which  aircraft  might  create  a  potential 
conflict  prior  to  an  alert  level  indication.  When  predictors  were  selected,  they  were  displayed  for 
all  aircraft  (see  Figure  9).  The  predictor  symbol  was  identical  to  the  shape  of  the  CDTI-AL  alert 
symbology  with  a  line  and  a  circle  that  represented  5  nm  in  diameter,  except  that  the  predictor 
symbology  was  white,  and  the  alert  symbology  was  blue.  Selected  predictor  time  was  displayed 
at  the  lower  right  hand  comer  of  the  navigation  display.  In  addition,  to  reduce  clutter,  predictors 
and  callsigns  of  the  non-conflicting  traffic  were  automatically  cleared  from  the  display  at  the 
onset  of  a  CDTI-AL  alert  but  could  be  reselected  at  any  time. 
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Figure  9.  CDTI-AL  depicting  predictors  selected. 


Finally,  pilot  participants  could  also  de-clutter  the  navigation  display  by  changing  the  horizontal 
map  range.  Ranges  available  were  similar  to  those  available  on  the  navigation  display  on  most 
Boeing  747-400  aircraft  (10, 20, 40,  80, 160,  320,  and  640  nm). 

2.3. 2. 3  PAS  Laboratory  and  Intruder  Aircraft  Simulator 

The  PAS  laboratory  at  NASA  ARC  was  the  same  system  employed  by  the  WJHTC.  NASA 
ARC  had  one  workstation  configured  as  the  intruder  aircraft  simulator.  The  intruder  aircraft 
supplied  the  simulation  pilot  with  the  same  display  of  traffic  and  airborne  alerting  logic  as  the 
pilot  participants.  The  workstation  also  supplied  the  ZME  display  of  traffic. 

2. 3. 2.4  NASA  Audio  and  Video  Recording 

There  were  three  cameras  within  the  NASA  ARC  simulator  that  provided  views  of  the  flight 
deck  and  the  CDTI-AL  In  addition,  the  microphones  within  the  simulator  allowed  for 
recording  of  all  airoair  and  airoground  communications,  along  with  communications  within 
the  cockpit. 


^  Video  and  audio  recordings  were  used  to  provide  a  mechanism  to  explore  issues  that  may  have  been  unclear  in  the 
objective  and  subjective  data  obtained  during  this  simulation.  The  information  contained  on  these  tapes  was  not 
used  for  any  other  purpose.  All  tapes,  so  obtained,  are  held  by  NASA  ARC  and  were  made  available  only  to  the 
members  of  the  experiment  team. 
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2.4  Airspace 


Two  ZME  sectors,  sectors  44  and  21,  were  emulated  in  the  experiment.  All  adjacent  surrounding 
sectors  were  combined  on  a  single  position  and  collectively  referred  to  as  sector  78  (ghost 
sector).  Sector  78  was  staffed  by  an  experiment  team  member  for  realism.  Figure  10  depicts  the 
two  sectors  selected  for  the  study  and  all  adjacent  sectors.  ZME  airspace  was  chosen  because  it 
is  currently  one  of  the  locations  where  DSR  and  URET  have  been  operationally  fielded.  The 
sectors  were  selected  based  on  recommendations  from  ZME  personnel  for  the  following  reasons: 

•  Both  sectors  21  and  44  are  high  altitude  sectors  that  contain  moderate  to  high  traffic  flows 
producing  moderate  to  high  workload. 

•  The  sectors  are  considered  to  be  of  moderate  to  high  complexity  (subjectively  described  by 
ZME). 

•  The  sectors  are  adjoining,  therefore  presenting  the  opportunity  to  observe  inter-sector 
coordination. 

•  The  sectors  are  from  different  areas  of  specialization.  This  eased  the  impact  on  field  staffing 
while  they  participated  in  this  research  endeavor. 
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2.4.1  Sector  21  (Conway  High) 


Sector  21  (Conway  High)  is  a  ZME  high  altitude  sector  encompassing  the  airspace  between  FL 
240  and  FL310.  The  focal  point  for  the  route  segments  within  the  sector  is  the  Little  Rock  (LIT) 
Very  High  Frequency  Onrni  Directional  Radio  Range  (VOR)  Tactical  Air  Navigational  Aid 
(VORTAC).  This  is  the  only  navigational  aid  in  the  sector  and  approximately  1 1  jet  route 
segments  converge  over  it.  The  sector’s  location  and  roughly  rectangular  shape  is  such  that  it  is 
bounded  on  the  southwest  by  Fort  Worth  Center,  on  the  northwest  by  the  Razorback  High  sector, 
on  the  northeast  by  Blytheville  High  sector,  on  the  east  by  the  Memphis  High  sector,  and  on  the 
south  by  the  Pine  Bluff  High  sector.  The  sector  is  approximately  190  nm  from  northeast  to 
southwest  and  80  nm  from  north  to  south.  In  the  simulation,  the  sector  was  emulated  as 
described,  except  that  the  airspace  was  expanded  to  include  FL240  and  all  altitudes  above. 

2.4.2  Sector  44  (Pine  Bluff  Hiehl 

Sector  44  (Pine  Bluff  High)  is  a  ZME  high  altitude  sector  encompassing  the  airspace  of  FL240 
and  above.  The  sector  is  one  of  seven  high/ultra  high  sectors  within  the  ARTCC.  The  sector’s 
location  and  rectangular  shape  starts  southwest  of  LIT  VORTAC  and  proceeds  northeast  for  135 
miles,  turns  south-southwest  for  50  miles,  turns  southwest  for  100  miles  then  intercepts  the 
Dallas/Fort  Worth  ARTCC  boundary.  From  that  point,  the  sector  proceeds  northwest  for  50 
miles  to  the  point  of  its  beginning.  In  the  simulation,  the  sector  was  emulated  as  described. 

2.5  Experimental  Conditions 

The  simulation  experiment  consisted  a  four  conditions  defined  by  various  levels  of  controller  and 
flight  crew  shared-separation  responsibilities.  The  conditions  were  Current  Operations  (CO), 

CO  with  CDTI-AL  (CO:CDTI),  Shared-Separation  Level  1  (SS:L1),  and  Shared-Separation 
Level  2  (SS:L2).  Each  condition  used  a  different  set  of  procedures  that  reflected  their  changing 
roles  and  responsibilities.  Current  standard  separation  rules  of  5  nm  horizontal  or  1000/2000  ft 
vertical  as  appropriate  were  observed  for  all  conditions.  A  within-subjects  design  was  utilized 
where  all  flight  crew  and  controller  participants  were  exposed  to  each  condition. 

•  CO 

This  condition  represented  current  ATC  environment.  URET  was  operational.  Standard  air 
traffic  procedures  defined  in  the  Controller  Handbook  71 10.65  (FAA,  2000),  Federal 
Aviation  Regulations  (FAR)  Part  91  (FAA,  1997),  and  the  Aeronautical  Information  Manual 
were  applied  during  this  condition.  The  pilots  in  this  condition  did  not  have  access  to  the 
CDTI. 

•  CO:CDTI 

This  condition  emulated  elements  of  the  RTCA  (1995)  definition  of  the  free  flight 
environment.  This  condition  simulated  all  equipment  and  procedures  of  CO  with  the 
following  changes: 
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Pilots  had  access  to  a  CDTI-AL. 


Pilots  could  query  controllers  (e.g.,  regarding  potential  conflicts  or  traffic)  and  make 
requests  based  on  information  from  their  CDTI-AL  displays  to  maximize  efficiency  or 
for  safety  concerns. 

•  SS:L1 

This  condition  emulated  elements  of  the  RTCA  (1995)  definition  of  the  free  flight 

envirorunent.  This  condition  simulated  all  equipment  and  procedures  of  CO:CDTI  with  the 

following  changes: 

-  All  flight  crews  started  SS:L1  responsible  for  their  own  separation  (i.e.,  free  flight). 

-  All  initial  flight  plans  and  altitudes  were  considered  as  optimum  for  the  current 
conditions. 

-  Flight  crews  were  free  to  initiate  any  maneuver  (i.e.,  change  heading,  altitude,  speed,  or 
any  combination)  provided  they  first  inform  ATC. 

-  Flight  crews  were  able  to  communicate  with  other  flight  crews  on  the  air<->air  frequency. 
Controllers  could  monitor  the  air^air  frequency  as  desired,  but  it  was  not  required. 

-  Flight  crews  were  instructed  to  use  specific  right-of-way  rules  to  resolve  conflicting 
situations. 

-  Flight  crews  could  cancel  free  flight^  of  their  own  aircraft  at  any  time. 

-  Controllers  were  instructed  to  resolve  any  disputes  between  pilots. 

-  Controllers  were  instructed  to  issue  traffic  alerts^  to  the  aircraft  involved  in  a  URET  RED 
alert. 

Controllers  were  instructed  to  coordinate  all  traffic  alerts  on  aircraft  not  under  their 
control  with  the  controlling  sector.  Controllers  receiving  a  coordinated  traffic  alert  were 
instructed  to  forward  this  to  the  subject  aircraft  unless  that  aircraft  had  already  advised 
that  a  resolution  was  in  progress. 

-  Controllers  could  wait  to  issue  a  traffic  alert  until  the  subject  aircraft  was  under  their 
control. 


*  For  the  procedures  of  this  study,  the  cancellation  of free  flight  was  defined  as  the  cancellation  of  shared-separation 
operations  resulting  in  aircraft  separation  responsibility  switching  from  pilots  (air)  back  to  controllers  (ground). 

’  For  the  procedures  of  this  study,  traffic  alert  was  defined  as  an  advisory  that  an  aircraft  was  involved  in  a  URET 
RED  alert. 
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-  Controllers  could  only  cancel  free  flight  (for  one  or  a  pair  of  aircraft)  if  they  had  queried, 
or  had  knowledge  of  the  intentions  of,  at  least  one  of  the  aircraft. 

-  Sector-wide  cancellation  of  free  flight  was  NOT  allowed. 

-  To  issue  a  control  instruction  to  a  flight  crew,  controllers  were  instructed  to  first  cancel 
free  flight  for  that  aircraft. 

-  Flight  crews  whose  free  flight  had  been  canceled  remained  under  ATC  control 
unless/until  the  controller  resumed  free  flight. 

-  Only  controllers  could  resume  free  flight. 

•  SS:L2 

This  condition  emulated  elements  of  the  RTCA  (1995)  definition  of  the  free  flight 

environment.  This  condition  simulated  all  equipment  and  procedures  of  SS:L1  except  for  the 

following  changes: 

-  Flight  crews  were  not  required  to  inform  the  controller  before  initiating  any  maneuver. 

-  Controllers  were  not  required  to  issue  traffic  alerts  to  aircraft,  but  could  do  so.  (However, 
they  were  still  required  to  coordinate  all  URET  RED  alerts  on  aircraft  not  under  their 
control  with  the  controlling  sectors). 

-  Controllers  were  not  required  to  update  the  Host  for  altitude  or  flight  plan  amendments  in 
this  condition  (the  ADO  emulated  automatic  datalink  updates  to  the  Host). 

-  Controllers  could  not  cancel  free  flight  for  any  aircraft  at  any  time. 

Table  2  summarizes  characteristics  of  the  experimental  conditions. 


Three  base  scenarios  were  developed  from  flight  plans  extracted  from  ZME  System  Analysis  and 
Recording  (SAR)  tapes  and  accompanying  Adaptation  Control  Environment  System 
configuration  tapes  obtained  from  the  field.  ZME  personnel  assisted  in  the  development  and 
modification  of  the  scenarios. 
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Table  2.  Experimental  Condition  Characteristic  Summary 


Characteristics 

CO 

COtCDTI 

SS:L1 

SS:L2 

Separation  standards  of  5  nm  horizontal  or 

1 000/2000  ft  vertical 

V 

V 

V 

V 

URET  was  available  to  controllers 

V 

V 

V 

Controllers  coordinated  URET  red  alerts  with  other 
sectors 

V 

V 

V 

V 

Controllers  had  full  separation  responsibility 

V 

V 

Pilots  required  to  request  clearance  from  controllers 
prior  to  maneuvering 

V 

V 

_ 

CDTI-AL  was  available  to  pilots 

V 

V 

V 

Pilots  and  controllers  shared-separation 
responsibility 

V 

V 

Air4->air  frequency  was  available 

V 

V 

Pilots  used  right-of-way  rules  while  resolving 
potential  conflicts 

V 

V 

Pilots  could  cancel  free  flight 

V 

V 

Controllers  could  cancel  free  flight 

V 

Pilots  could  initiate  any  maneuver  but  were  required 
to  first  inform  controllers  prior  to  maneuvering 

V 

Controllers  were  required  to  issue  traffic  alerts  to 
aircraft  concerning  URET  red  alerts 

V 

Pilots  did  not  have  to  inform  controllers  prior  to 
maneuvering 

V 

The  data  allowed  for  the  realistic  representation  of  sector  boundaries,  jet  routes,  and  fixes  for  the 
simulated  sectors.  To  suit  simulation  needs,  the  traffic  scenarios  were  modified  by  altering  some 
traffic  flows,  creating  planned  conflicts,  and  by  increasing  the  number  of  aircraft  to  compensate 
for  other  complexity  limitations.  To  control  extraneous  factors  in  the  evaluation,  the  traffic 
scenarios  did  not  include  severe  weather,  and  emergency  or  critical  situations.  In  addition,  there 
were  very  few  ascending  or  descending  aircraft  to  particularly  avoid  excessive  URET-muted 
alerts  that  might  detract  from  the  evaluation. 
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One  of  the  base  scenarios  was  used  to  create  ATC  training  runs.  The  other  two  base  scenarios 
were  used  to  create  data  collection  runs.  These  two  base  scenarios  were  slightly  modified*  to 
create  four  unique  traffic  situations  for  the  four  conditions  presented  to  the  controllers  and  pilots. 
Specifically,  CO  and  SS:L1  were  created  from  the  same  base  scenario,  and  CO:CDTI  and  SS:L2 
were  created  from  the  same  base  scenario. 

2.5. 1.1  Traffic  Scenario  Characteristics 


The  experiment  consisted  of  three  training  runs  and  four  data  collection  runs  that  reflected 
varying  levels  of  ATC  and  shared-separation  operations.  The  scenario  used  to  create  the  ATC 
training  runs  was  45  minutes  long.  The  scenarios  used  to  create  the  data  collection  runs  were 
100  minutes  long.  As  traffic  began  to  build  (approximately  the  10*'’  minute),  participant 
controllers  were  given  sector  briefings  and  asked  to  start  their  participation  in  the  simulation. 
During  the  building  period,  scripted  events  were  limited  to  routine  pilot-to-controller  interactions 
(e.g.,  climb  or  descend  requests  and  direct  routing  requests),  and  when  appropriate,  routine  pilot- 
to-pilot  interactions  (e.g.,  requests  for  information  and  coordination  of  maneuvers).  By  design, 
there  were  no  pilot-to-pilot  interactions  during  CO  and  CO:CDTI  conditions.  Table  3  describes 
traffic  scenario  characteristics. 


Table  3.  Traffic  Scenario  Characteristics 


Experimental 

Condition 

Traffic  Volume 

Duration 

(minutes) 

Purpose 

ATC 

CDTI-AL 

COiCDTI 

Moderate 
to  High’ 

N/A 

45 

ATC 

Training 

SS:L1 

Moderate 
to  High 

N/A 

45 

ATC 

Training 

SS:L2 

Moderate 
to  High 

N/A 

45 

ATC 

Training 

CO 

Moderate 
to  High 

100 

Data 

Collection 

COtCDTI 

Moderate 
to  High 

High 

100 

Data 

Collection 

SS:L1 

Moderate 
to  High 

High 

100 

Data 

Collection 

SS:L2 

Moderate 
to  High 

High 

100 

Data 

Collection 

*  All  aircraft  callsigns  and  the  destination  airports  for  the  conflict  aircraft  were  changed. 

’  ATC  moderate-to-high  traffic  volume  was  emulated  as  greater  than  16  aircraft  for  sector  21  control,  and  greater 
than  13  aircraft  for  sector  44  control. 

CDTl-AL  high  traffic  volume  was  emulated  at  1 5  or  greater  aircraft  visible  on  the  display. 
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2.5. 1.2  Planned  Conflicts  and  NASA  20-Minute  Flight  Segments 


Conflict  detection  and  resolution  is  an  integral  part  of  air  traffic  control.  To  assist  in  the 
evaluation  of  shared-separation  operations,  planned  conflicts  between  aircraft  occurred  in  each 
run.  All  planned  conflicts  involved  two  aircraft  converging  at  an  acute  angle. 

ATC  training  runs  had  eight  plaimed  conflicts  between  simulated  aircraft  (the  NASA  ARC 
simulator  and  intruder  aircraft  did  not  join  these  runs  as  they  had  independent  training).  Each  of 
the  four  data  collection  runs  (CO,  CO:CDTI,  SS:L1,  and  SS:L2)  had  16  planned  conflicts.  Eight 
2-aircraft  conflicts  of  similar  complexity  were  plaimed  in  each  sector.  In  sector  21,  the  NASA 
ARC  simulator  and  intruder  aircr^  pair  were  involved  in  three  out  of  the  eight  planned 
conflicts.  Once  a  conflict  was  resolved  between  them  and  the  aircraft  moved  out  of  the  sector, 
the  simulators  rejoined  the  run  as  different  aircraft  with  different  call  signs.  Each  of  these  three 
“flight  segments”  for  the  NASA  ARC  simulator  and  intruder  aircraft  lasted  approximately  20 
minutes.  In  sector  44,  WJHTC  simulation  pilots  flew  all  planned  conflict  aircraft  pairs. 

2.5.2  Pilot  Right-of-Wav  Rules 

During  SS:L1  and  SS:L2  runs,  pilot  participants  and  simulation  pilots  were  instructed  to  use 
FAR  Part  91  right-of-way  rules  (when  possible)  while  resolving  their  own  conflicts.  Although 
the  right-of-way  rules  are  normally  only  applicable  during  visual  meteorological  conditions,  it 
has  been  suggested  that  they  may  be  applied  in  shared-separation  operations  to  help  guide 
negotiations  (RTCA,  1995).  All  planned  conflicts  involving  simulation  pilots  were  scripted  (by 
the  rules)  for  resolution  action  and  communications.  The  pilot  right-of-way  rules  were  as 
follows: 

•  The  aircraft  on  the  right  had  the  right-of-way. 

•  The  aircraft  being  overtaken  had  the  right-of-way. 

•  Aircraft  that  were  converging  head-on  each  should  have  altered  course  to  the  right. 

•  During  most  conflict  situations,  the  aircraft  that  did  not  have  the  right-of-way  should  have 
initiated  the  communication  with  the  aircraft  that  had  the  right-of-way. 

2.5.3  Phraseology 

Except  for  the  pilot  participants,  all  simulation  pilot  phraseology  was  scripted.  The  pilot 
participants  were  instructed  to  use  the  phraseology  that  they  would  use  based  on  their  current 
procediues  in  domestic  airspace.  The  only  specific  instruction  provided  to  the  flight  crews 
regarding  phraseology  pertained  to  free  flight  cancellation.  If  the  flight  crew  wanted  to  terminate 
free  flight,  they  were  instructed  to  state  their  aircraft  identifier  (ACID)  and  specifically  indicate 
to  the  controller  that  they  wanted  to  “cancel  free  flight.”  flight  crews  were  also  told  that  they 
could  contact  any  other  aircraft,  but  they  were  not  given  any  phraseology  recommendations  for 
those  communications.  In  addition  to  the  standard  phraseology  as  described  in  the  FAA  Order 
71 10.65  (FAA,  2000),  controllers  were  trained  on  additional  simulation  phraseology  (see 
Table  4). 
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Table  4.  Additional  Controller  Simulation  Phraseology 


Action 

Phraseology 

Condition 

Cancellation  of  Free 
Flight 

Controller  -  “yfC/D  (and  ACID),  free  flight 
canceled'  and  issue  the  appropriate  control 
action. 

SS:L1 

Resumption  of  Free 
Flight 

Controller  -  '‘'’ACID  (and  ACID),  resume  free 
flight:' 

SS:L1 

Acknowledge  Pilot 
intentions 

Pilot  -  informs  controller  of  an  intended 
maneuver. 

Controller-  “ACID,  roger” 

SS:L1 

Aircraft  coordination 

Controller  -  ''“‘ACID,  traffic  alert  with  ACID  at 

SS:L1  and 

for  RED  URET  alerts 

(altitude)  at  (time),  advise  intentions  ” 

SS:L2 

Sector  coordination  for 
RED  URET  alerts 

Controller  Sector  1  -  “Traffic  alert  ACID’’ 

Controller  Sector  2  -  “Go  ahead' 

Controller  Sector  1  -  “ACID  with  ACID  at 
altitude  at  time" 

SS:L1  and 
SS:L2 

2.5.4  Frequencies 

In  addition  to  land  lines  and  sector  frequencies,  the  experiment  emulated  an  air<-»air  frequency. 
The  airoair  frequency  was  provided  to  avoid  frequency  congestion  problems  and  was  only 
available  during  SS:L1  and  SS;L2  conditions.  On  the  air<->air  frequency,  the  pilots  were  able  to 
communicate  among  themselves  and  negotiate  resolution  strategies.  Controllers  were  able  to 
selectively  monitor  the  frequency  as  desired  but  were  not  permitted  to  transmit  on  the  frequency. 
Table  5  provides  the  frequencies  used  in  this  experiment. 
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2.7.2  WJHTC  Simulation  Activities 


Each  group  of  controllers  and  the  EOs  participated  for  a  3 -day  simulation  period  (from  12  to  8 
p.m.  EST).  A  daily  schedule  for  controller  participants  is  provided  in  Appendix  A. 

On  the  first  day,  they  were  provided  an  experiment  briefing.  Following  the  initial  briefing, 
controllers  were  assigned  to  a  sector  (21  or  44)  and  position  (R-side  or  D-side),  which  remained 
constant  throughout  the  experiment.  The  EOs  were  also  assigned  to  a  sector.  Laboratory 
familiarization  and  hands-on  training  followed. 

On  the  second  day,  controllers  and  EOs  were  provided  additional  hands-on  training.  Following 
the  completion  of  all  training,  data  collection  runs  started.  At  the  beginning  of  each  run, 
participants  were  again  briefed  on  the  procedures  of  the  particular  condition  they  were  about  to 
experience.  They  were  also  provided  with  an  aid  chart  on  their  control  position  describing  the 
key  procedures  for  the  run.  During  each  run,  the  EOs  watched  sector  operations  and  recorded 
interesting  and  critical  events. 

Data  collection  runs  continued  through  the  third  day.  At  the  end  of  all  data  collection  runs,  the 
experiment  team  held  a  semi-structured  group  debriefing  session.  The  purpose  of  this  debriefing 
was  to  provide  an  opportunity  to  share  information  that  was  not  captured  in  the  forms. 

Participation  in  this  study  was  strictly  voluntary,  and  the  privacy  of  all  participants  was  and  will 
be  protected.  Strict  adherence  to  all  federal,  union,  and  ethical  guidelines  was  maintained 
throughout  the  study. 

2.7.2. 1  Controller  Briefing 

Members  of  the  experiment  team  briefed  the  participants  in  a  classroom  setting  prior  to  entering 
the  laboratory  area.  The  participants  were  encouraged  to  ask  questions.  The  participants  were 
also  provided  with  the  briefing  materials  contained  in  Appendix  B. 

The  briefing  covered  the  following  topics: 

•  Human  Research  Minimal  Risk  Consent  Document, 

•  Participant’s  role  in  the  study, 

•  Study  objectives, 

•  Study  methodology, 

•  Airspace  structure, 

•  Aircraft  equipage  and  procedures, 

•  Air  traffic  characteristics, 

•  Laboratory  equipment  and  configuration,  and 

•  Rules  and  procedures. 
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Following  the  briefing,  the  participants  were  requested  to  complete  the  Background  Information 
Form  (Appendix  C)  and  the  Human  Research  Minimal  Risk  Consent  Document  (Appendix  D). 

2.1.22  Laboratory  Familiarization 

Although  the  PF  was  configured  to  replicate  ZME  sectors  44  and  21  with  high  fidelity, 
differences  between  the  field  and  the  laboratory  configuration  existed.  All  differences  were 
briefed  in  detail  and  instructions  on  equipment  usage  were  provided.  Equipment  training  and 
laboratory  familiarization  lasted  approximately  2  hours. 

2.7.2.3  Controller  Training 

Three,  45-minute  training  runs  were  provided  to  allow  participants  to  gain  experience  with 
shared-separation  operations  and  provide  additional  practice  with  the  laboratory  equipment*^. 
Members  of  the  experiment  team  and  EOs  were  available  throughout  training  to  answer 
questions.  A  15-minute  classroom  group  discussion  followed  each  training  run. 

2.7.2.4  Experimental  Condition  Run  Order 

Following  the  training  runs,  data  collection  activities  started.  All  participant  groups  (controller 
and  pilot)  participated  in  all  four  runs.  The  order  of  condition  presentation  was  counter-balanced 
across  the  four  data  collection  groups.  Table  6  depicts  the  order  of  condition  presentation. 


Table  6.  Run  Order  by  Group 


Group  1 

Group  2 

Group  3 

Group  4 

Run  1 

CO 

CO:CDTI 

SS:L2 

SS:L1 

Run  2 

SS:L2 

SS:L1 

CO 

CO:CDTI 

Run  3 

CO;CDTI 

CO 

SS:L1 

SS:L2 

Run  4 

SS:L1 

SS:L2 

CO:CDTI 

CO 

2.7.3  NASA  ARC  Pre-Simulation  Activities 

In  the  weeks  prior  to  simulation,  the  intruder  simulation  pilot  was  trained  to  assure  operationally 
consistent,  accurate,  and  timely  responses  to  controller  instructions  and  requests.  In  addition,  the 
simulation  pilot  was  instructed  on  how  to  provide  operationally  realistic  air<->air 
communications  in  a  shared-separation  context.  The  following  topics  were  discussed: 


Due  to  independent  training  sessions,  the  NASA  ARC  simulator  and  intruder  aircraft  did  not  participate  in  these 
training  scenarios. 
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•  Study  objectives, 

•  Study  methodology, 

•  Airspace  structure, 

•  Air  traffic  characteristics, 

•  Aircraft  equipage, 

•  Controller  procedures, 

•  Anticipated  controller  actions  during  shared-separation  op)erations,  and 

•  Anticipated  flight  crew  comments  and  maneuvering  during  shared-separation  operations. 

The  intruder  simulation  pilot  participated  in  trial  runs  of  all  conditions  prior  to  the  beginning  of 
data  collection.  The  trials  included  air4->ground  communications,  airoair  communications,  and 
maneuvering  of  the  simulator  aircraft. 

2.7.4  NASA  ARC  Simulation  Activities 

Each  flight  crew  participated  in  the  study  for  two  8-hour  days  (from  9  to  5  p.m.  EST).  A  daily 
schedule  for  pilot  participants  is  provided  in  Appendix  E. 

On  the  morning  of  the  first  day,  the  pilot  participants  were  given  an  experiment  briefing  and 
training  on  the  tasks  they  would  perform  during  the  experiment.  All  pilot  participants  were 
qualified  on  the  Boeing  747-400  aircraft  type;  therefore,  no  training  or  familiarization  with  the 
simulator  was  required,  with  the  exception  of  the  new  flight  deck  tools  provided  for  shared- 
separation. 

Data  collection  runs  began  on  the  afternoon  of  the  first  day.  Each  condition  (CO,  COrCDTI, 
SS:L1,  and  SS:L2)  had  three  flight  segments.  At  the  end  of  the  three  flight  segments 
representing  a  condition,  a  form  was  given  to  address  questions  regarding  tasks  and  workload 
during  that  condition.  At  the  end  of  all  data  collection,  pilot  participants  completed  two  more 
forms  and  were  debriefed  to  obtain  all  their  feedback  relative  to  the  goals  and  tasks  of  the  study. 
See  Appendix  F  for  these  pilot  forms. 

2.7.4. 1  Flight  Crew  Briefing 

Flight  crews  participated  in  a  90-minute  briefing  covering  the  general  goals  of  the  free  flight 
concept  and  this  study.  The  briefing  emphasized  the  potential  for  increased  operational 
flexibility  and  efficiency  with  free  flight.  Topics  in  the  briefing  also  included 

•  the  Human  Research  Minimal  Risk  Consent  Document; 

•  roles  of  pilots  and  controllers  in  this  study; 

•  new  display  features,  alerting  logic,  and  underlying  technology  assumptions; 

•  maneuvering  and  communication  procedures,  including  pilot  right-of-way  rules; 

•  rules  governing  free  flight  for  each  applicable  condition; 
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•  general  description  of  flight  information  (e.g.,  destination  and  path);  and 

•  safety  briefing  for  NASA  ARC  simulator. 

Following  the  briefing,  the  participants  were  requested  to  complete  the  Human  Research 
Minimal  Risk  Consent  Document  contained  in  Appendix  D. 

2.7.4.2  Flight  Crew  Training 

Following  the  briefing,  pilots  participated  in  approximately  90  minutes  of  training  in  the  NASA 
ARC  simulator.  The  flight  crew  training  runs  were  different  from  the  actual  experimental  runs, 
but  they  exposed  the  participants  to  the  different  conditions  and  procedures  under  which  the 
crews  were  expected  to  operate.  The  crews  were  provided  with  a  sense  of  the  timing  parameters 
associated  with  the  alerting  logic  and  had  an  opportunity  to  practice  the  verbal  procedures  used 
in  air<-»air  and  airoground  communication.  Questions  by  the  flight  crew  participants  were 
encouraged  during  both  the  briefing  and  training  sessions. 

1.1  A3  Experimental  Condition  Run  Order 

All  participant  groups  (controller  and  pilot)  participated  in  all  four  runs.  Each  flight  crew  flew 
three  20-miniute  fi^t  segments  per  run.  The  order  of  condition  presentation  was  random  and 
counter-balanced  across  the  four  data  collection  groups. 

2.8  Data  Collection 

Subjective  and  objective  data  were  collected  throughout  the  study  from  the  ground-side 
(participant  controllers,  EOs,  and  the  ATC  environment),  and  the  air-side  (pilot  participants  and 
the  flight  deck). 

2.8.1  Ground-Side  Subjective  Data 

Subjective  data  were  collected  from  controllers  and  EOs  via  forms  (Appendix  C),  interval 
workload  ratings,  and  debriefing  sessions. 

2.8. 1.1  Form  Data 

Participant  controllers  and  EOs  completed  the  Background  Information  Form  immediately  after 
the  initial  briefing  session  at  the  WJHTC.  The  background  forms  solicited  information  related  to 
professional  experience  and  other  relevant  information. 

EOs  also  completed  the  During-the-Run  Form  throughout  the  simulation.  Using  this  form,  the 
EOs  recorded  critical  events,  free  flight  cancellations,  controller  actions,  and  observations  related 
to  the  impact  of  conflicts  and  shared-separation  operations. 

At  the  end  of  each  run,  both  the  participant  controllers  and  EOs  completed  post-run  forms.  The 
Controller  Post-Run  Form  solicited  information  regarding  the  traffic,  simulation  environment, 
workload  ratings,  and  impact  of  conflicts  and  shared-separation  operations.  The  EO  Post-Run 
Form  solicited  information  regarding  the  overall  workload  and  impact  of  conflicts  and  shared- 
separation  operations. 
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At  the  end  of  all  runs,  the  participant  controllers  completed  an  Exit  Form.  This  form  elicited 
information  regarding  simulation  fidelity,  adequacy  of  training  for  simulation,  automation  needs, 
and  the  effects  of  shared-separations  operations. 

2.8. 1.2  Interval  Workload  Data 

During  each  run,  the  participant  controllers  rated  their  instantaneous  workload  (combined 
cognitive  and  physical)  on  a  l-to-5  scale  (1  =  very  low,  3  =  moderate,  and  5  =  very  high),  at 
5-minute  intervals,  using  WAKs.  In  the  few  instances  when  the  laptop  and/or  WAK  did  not 
function  properly,  researchers  supplied  paper  forms  to  participant  controllers  to  record  workload 
ratings. 

The  following  instructions  were  given  to  participant  controllers  regarding  interval  workload 
ratings: 

•  The  WAK  will  illuminate  and  sound  a  small  beep  at  every  5-minute  interval.  At  that  time, 
you  are  requested  to  press  a  key  corresponding  to  your  instantaneous  workload  level. 

•  When  reporting  your  workload  rating,  please  consider  both  cognitive  and  physical  workload, 

•  The  workload  rating  scale  was  as  follows: 


1 

2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

The  operational  meaning  of  the  rating  scale  was  explained  as  follows: 

Rating  Operational  Meaning 

1  Your  workload  is  very  low  and  you  can  complete  all  tasks. 

2  Your  workload  is  rather  low  and  there  is  little  chance  for  an  error  in  your  tasks. 

3  Your  workload  is  moderate  and  there  is  an  increasing  chance  of  error  in  your  tasks. 

4  Your  workload  is  rather  high  and  there  is  some  chance  for  an  error  in  your  tasks. 

5  Your  workload  is  very  high  and  you  may  have  to  leave  some  tasks  incomplete. 
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2.8. 1.3  Conflict  Detection  and  Resolution  Measures 

The  participant  controllers  were  asked  to  provide  different  points  of  conflict  detection  and 
resolution  as  follows: 

Point  A  -  Conflict  Detection  Point:  The  first  point  was  collected  for  all  four  experimental 
conditions  and  was  recorded  at  the  point  when  controllers  first  detected  a  potential  conflict 
between  aircraft  with  reasonable  certainty. 

Point  B  -  Conflict  Resolution  Point  under  Current  Operations:  The  second  point  was  collected 
only  for  SS:L1  and  SS:L2.  The  second  point  was  recorded  at  the  point  when  controllers  would 
have  taken  action  to  resolve  a  potential  conflict  under  current  operating  rules  (as  described  in 
71 10.65  [FAA,  2000]).  Controllers  were  asked  to  assume  that  not  all  flight  crews  have 
CDTI-AL. 

Point  C  -  Conflict  Resolution  Point  under  SS:L2  Operations:  The  third  point  was  collected  only 
for  SS:L2.  The  third  point  was  recorded  when  controllers  would  have  t^en  action  to  resolve  a 
potential  conflict  under  SS:L2  conditions  where  all  flight  crews  have  CDTI-AL  and  are 
responsible  for  separation. 

The  EOs  recorded  these  points  for  each  sector  and  noted  the  time  on  the  Observer  During-the- 
Rxm-Form.  For  Point  A,  the  EOs  also  noted  whether  the  controller  used  URET. 

Results  and  subsequent  analyses  of  these  data  were  intended  to  address  controller  situation 
awareness.  However,  there  were  problems  with  the  automated  tool  used  to  collect  these  data 
resulting  in  unacceptable  levels  of  accuracy.  Therefore,  results  and  discussion  about  these  data 
are  not  included  in  the  report. 

2.8. 1.4  Debriefing  Sessions 

Semi-structUred,  debriefing  sessions  were  conducted  with  each  controller  group  and  the  EOs. 
The  controllers  and  EOs  were  given  the  opportunity  to  provide  any  additional  information  about 
their  experiences  in  the  simulation,  their  thoughts  about  the  concepts,  procedures  and  tools 
investigated  in  this  project,  and  to  have  the  researchers  answer  any  remaining  questions.  All 
forms  provided  blank  spaces  for  participants  to  provide  open-ended,  descriptive  information  and 
comments. 

The  debriefings  were  recorded  on  audiocassettes.  Table  7  summarizes  subjective  data  that  were 
collected  during  the  simulation. 
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Table  7.  Ground-Side  Subjective  Data 


Instrument 

Users 

Completed 

Objective 

Background  Form 

Controllers 

EOs 

Once  before 
first  training  run 

Gather  demographic 
information. 

During-the-Run 

Form 

EOs 

During  each  run 

Record  critical  and  interesting 
events. 

Interval  Workload 
Ratings 

Controllers 

Every  5  minutes 
during  each  run 

Electronically  record  controller 
workload  ratings  using  WAK. 

Point  A  -  Conflict 
Detection 

Controllers 

During  each  run 

Record  on  the  During-the-Run 
Observer  Form. 

Point  B  -  Conflict 
Resolution 

Controllers 

During  SS:L1 
and  SS:L2  runs 

Record  on  the  During-the-Run 
Observer  Form. 

Point  C  -  Conflict 
Resolution 

Controllers 

During  SS:L2 
runs 

Record  on  the  During-the-Run 
Observer  Form. 

Post-Run  Form 

Controllers 

After  each  run 

Elicit  controller  comments  and 
ratings  related  to  the  conflict 
situations,  communications, 
shared-separation,  scenario 
information,  workload, 
situation  awareness,  and  so  on. 

Post-Run  Form 

EOs 

After  each  run 

Record  EO  observations 
related  to  conflicts, 
communications,  shared- 
separation,  workload,  and  so 
on. 

Exit  Form 

Controllers 

End  of  all  runs 

Gather  information  regarding 
impact  of  shared-separation 
and  conflicts  on  workload, 
automation  needs,  simulation 
training  adequacy  and  fidelity. 

De-briefing 

EOs 

Controllers 

End  of  all  runs 

Collective  discussions  of 
shared-separations  and  gather 
ground-side  information  that 
was  not  previously  acquired. 
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2.8.2  Ground-Side  Objective  Data 


Objective  data  related  to  URET  alerts  and  trial  plans,  voice  communication  data.  Host  data,  and 
audio  and  video  data  were  collected. 

•  URET  Alerts  and  Trial  Plans 

The  number  and  duration  of  URET-reported  red  and  yellow  alerts  were  extracted  from 
the  URET  data  logs.  The  frequency  of  URET  trial  plans  was  also  extracted  from  the 
URET  log. 

•  Push-to-Talk  Transmission  Data 

The  number  and  duration  of  ground— >air  and  land  line  push-to-talk  transmissions  (PTTs) 
were  recorded.  A  ground->air  PTT  is  defined  as  a  verbal  message  from  the  controller  to 
the  pilot,  the  duration  of  which  is  measured  from  the  onset  to  the  end  of  a  key  press. 

•  Host  Data 

The  number  of  aircraft  at  every  five-minute  interval,  the  total  number  of  aircraft  in  each 
run,  separation  violations,  MSD  data,  and  conflict  alerts  reported  by  the  Host  were 
extracted  from  SAR  tapes. 

•  Audio  and  Video  Recordings 

Each  run  was  video  and  audio  recorded  to  capture  the  interaction  between  controllers. 
The  purpose  was  to  gather  supplemental  data  to  assess  workload  levels  and  to 
substantiate  other  subjective  and  objective  data.  Audio  recordings  captured  the  ambient 
conversations  between  controllers  and  all  simulation  frequencies.  Video  recordings 
captured  general  views  of  the  sectors  and  the  URET  displays.  For  each  run,  the  video  and 
audio  were  recorded  onto  four  tapes  (two  tapes  per  sector).  Table  8  summarizes  the 
audio  and  video  recordings. 

Table  8.  ATC  Audio  and  Video  Recording 


Tape 

Video  View 

Left  Audio 
Communications 

Right  Audio 
Communications 

1 

S44  Overview 

S44  air<->ground, 
R-side/D-side  ambient 

land  line, 
R-side/D-side 
ambient 

2 

S21  Overview 

S21  airo^ground,  land  line 

S21  air4->ground, 
R-side/D-side 
ambient 

3 

S44 

URET  Display 

S44  air<->ground,  land  line 
R-side/D-side  ambient 

Ghost  Controller 
airoground 

4 

S21 

URET  Display 

All  Sector  21  audio  channels, 
R-side/D-side  ambient 

air4->air 
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Table  9  lists  objective  data  that  were  collected  during  each  run  during. 


Table  9.  Ground-Side  Objective  Data  Summary 


Data 

Source  of  Data 

Number  and  duration  of  ground^air  PTT 

PTT  recordings 

Number  and  duration  of  land  line  PTT 

PTT  recordings 

Peak  traffic  count 

SAR  tapes 

Traffic  density 

SAR  tapes 

MSD  of  all  planned  conflicts  (except  NASA 

ARC  simulator  conflicts) 

SAR  tapes 

Number  of  URET  trial  flight  plans 

URET  log 

Number  and  duration  of  URET  red  and  yellow 
conflict  alerts 

URET  log 

Number  and  duration  of  separation  violations 

SAR  tapes 

Number  of  Free  Flight  cancellations  by 
controllers 

Observer  and  controller  forms 

Number  of  Free  Flight  cancellation  requests  by 
pilots 

Observer  and  controller  forms 

2.8.3  Air-Side  Subjective  Data 

Subjective  data  were  collected  from  the  individual  pilot  participants  primarily  through  post-run 
and  exit  forms.  Additional  data  were  collected  during  a  debriefing  session  and  workload  data 
were  collected  following  each  flight  segment. 

2.8.3. 1  Form  Data 

Following  each  flight  segment,  each  pilot  participant  provided  a  rating  of  overall  workload  for 
the  20-minute  flight  segment.  Workload  ratings  were  collected  using  a  5-point  Likert  scale. 

Following  each  run,  which  consisted  of  three  20-min  flight  segments,  the  pilot  participants 
completed  a  Post-Run  Form  (Appendix  F).  This  form  assessed  various  aspects  of  pilots’ 
experiences  with  the  procedures  and/or  tools  used  during  that  run,  such  as  ratings  of  workload, 
situation  awareness,  safety,  and  effectiveness  of  the  new  flight  deck  tools. 
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Upon  completion  of  the  experiment,  pilot  participants  were  given  an  Exit  Form  (Appendix  F). 
This  form  collected  pilots’  demographic  information,  as  vk^ell  as  assessing  simulation  fidelity  and 
integrity  and  the  pilots’  overall  evaluation  of  tools  and  procedures  used  in  the  study. 

2.8.3.2  Debriefing  Sessions 

Debriefing  sessions  were  conducted  with  each  flight  crew.  The  flight  crews  were  given  the 
opportunity  to  provide  any  additional  information  about  their  experiences  in  the  simulation,  their 
thoughts  about  the  concepts,  procedures  and  tools  investigated  in  this  project,  and  to  have  the 
researchers  answer  any  remaining  questions.  All  forms  provided  blank  spaces  for  participants  to 
provide  open-ended,  descriptive  information  and  comments.  Table  10  summarizes  subjective 
data  that  were  collected  during  the  simulation. 

Table  10.  Flight  Crew  Subjective  Data 


Instrument 

Completed 

Objective 

Post-Flight  Workload 
Rating 

After  each  flight 
segment 

Assess  pilot  participants’  ratings  of 
overall  workload  across  the  preceding 
20-min  flight  segment. 

Post-Run  Form 

After  each  run 

Assess  pilot  participants’  ratings  of  the 
impact  of  the  tools  and/or  procedures  on 
safety,  information  requirements, 
workload,  situation  awareness,  and  so 
on  used  during  that  run. 

Exit  Form 

End  of  all  runs 

Gather  information  regarding  impact  of 
shared-separation  and  aircraft  conflicts 
on  workload,  automation  needs, 
simulation  training  adequacy,  and 
fidelity  across  entire  experiment. 

Debriefing 

End  of  all  runs 

Gather  any  additional  information 
concerning  the  pilots’  experiences  in  the 
simulation  and  to  discuss  the  concepts, 
procedures,  and  tools  under  study. 

2.8.4  Air-Side  Objective  Data 

Objective  data  were  collected  during  each  flight  segment  from  the  NASA  ARC  simulator 
computer  systems,  including  data  on  use  of  various  aircraft  flight  systems,  use  of  flight  deck 
display  controls,  and  communication  equipment.  Video  and  audio  recordings  of  flight  crew 
interactions  on  the  flight  deck  were  also  collected. 
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2.8.4. 1  NASA  ARC  Simulator  Data 


The  MSD,  number  of  separation  violations,  free  flight  cancellation  distances,  maneuver  type  and 
timing,  navigation  display  information,  and  CDTI-AL  parameters  relevant  to  the  NASA  ARC 
simulator  were  collected. 

2.8.4.2  Audio  and  Video  Recordings 

Each  flight  segment  was  audio  and  video  recorded  to  capture  the  interactions  between  the  pilot 
participants  and  their  use  of  the  aircraft  systems  as  well  as  the  communication  exchanges  with 
controllers  and  all  simulation  pilots.  Video  and  audio  were  recorded  onto  two  tapes.  Table  1 1 
depicts  the  recording  descriptions.  These  recordings  served  as  source  data  for  several  of  the 
variables  described  in  Table  12. 

Table  11.  Air-Side  Audio  and  Video  Recordings 


Tape 

Video  View 

Left  Audio 

Communications 

Right  Audio 
Communications 

1 

Quadrant  A:  Captain’s 
Primary  Flight  Display 

ambient  flight  deck 
communication 

selected  frequencies, 

(e.g.  air<->ground, 
air<->air) 

Quadrant  B:  Captain’s 
Navigation  Display 

ambient  flight  deck 
communication 

selected  frequencies, 

(e.g.  air<^ground, 
air<->air) 

Quadrant  C:  Captain’s 

Flight  Management 

Computer  (FMC)/Multi- 
Function  Control  Display 

Unit  (MCDU) 

ambient  flight  deck 
communication 

selected  frequencies, 

(e.g.  air<->ground, 
airoair) 

Quadrant  D:  First  Officer’s 
FMC/MCDU 

ambient  flight  deck 
communication 

selected  frequencies, 

(e.g.  air4->ground, 
air<->air) 

2 

Captain’s  Navigation 

Display 

ambient  flight  deck 
communication 

selected  frequencies, 

(e.g.  air<->ground, 
air^-^air) 
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Table  12.  Air-Side  Objective  Data  Summary 


Data 

Source  of  Data 

Number  and  duration  of  air^^-^ground  communications 
involving  NASA  ARC  simulator 

Video/audio  recordings 

Number  of  air4->air  communications  involving  NASA 
ARC  simulator 

Video/audio  recordings 

MSD  for  all  conflicts  involving  NASA  ARC  simulator 

NASA  ARC  simulator  output  data 

Number  of  separation  violations  for  all  conflicts 
involving  NASA  ARC  simulator 

NASA  ARC  simulator  output  data 

Number  of  free  flight  cancellations  by  controller  on 
conflicts  with  NASA  ARC  simulator 

Video/audio  recordings 

Number  of  free  flight  cancellation  requests  by  pilot 
participants 

Video/audio  recordings 

Distance  between  NASA  ARC  simulator  and  intruder 
aircraft  at  time  of  free  flight  cancellation 

NASA  ARC  simulator  output  data 

Type  of  maneuvers  made  by  NASA  ARC  simulator 
and  intruder  aircraft 

NASA  ARC  simulator  output  data  and 
video  recordings 

Type  of  maneuvers  issued  by  controllers  to  NASA 

ARC  simulator  and  intruder  aircraft 

NASA  ARC  simulator  output  data  and 
video  recordings 

Timing  of  first  maneuver  made  by  NASA  ARC 
simulator  and/or  intruder  aircraft  for  conflict 
resolution 

NASA  ARC  simulator  output  data  and 
video  recordings 

Frequency  and  timing  of  CDTI-AL  alerts  as  displayed 
on  the  navigation  display  of  NASA  ARC  simulator 

NASA  ARC  simulator  output  data  and 
video  recordings 

Conflict  detection  times  for  pilot  participants  for 
conflicts  involving  NASA  ARC  simulator 

Video/audio  recordings 

Time  spent  at  each  map  range  level  on  navigation 
display  of  NASA  ARC  simulator 

NASA  ARC  simulator  output  data 
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3.  RESULTS 


Within  each  sub-section,  the  order  of  data  results  corresponds  to  the  three  primary  objectives  of 
the  study. 

Due  to  the  exploratory  nature  of  the  research  and  small  sample  sizes,  inspection  of  the  mean  and 
±1  standard  error  of  the  mean  (SEM)  was  used  as  the  primary  method  to  analyze  the  data.  In 
general,  if  the  ±1  SEM  bars  overlapped,  then  the  two  means  were  considered  the  same.  If  they 
did  not  overlap,  then  the  means  were  reported  as  appearing  different.  The  Analysis  of  Variance 
(ANOVA)  statistical  method  was  sometimes  used  to  provide  additional  insight  for  future  areas  of 
research,  particularly  for  the  ground-side  data.  In  addition,  the  researchers  recognized  that  the 
rating  scales  used  in  this  study  reflect  ordinal  data  that  typically  warrants  analysis  using 
descriptive  statistics  such  as  median  and  range.  However,  in  order  to  provide  comparisons  to 
earlier  studies  and  make  the  results  more  understandable  to  a  broad  audience,  more  popular 
measures  such  as  mean  and  standard  deviation  (SD)  were  reported. 

3.1  Ground-Side  Results 


Most  of  the  ground-side  data  are  summarized  by  the  four  conditions  (CO,  COrCDTI,  SS:L1,  and 
SS:L2)  of  the  study.  Means  (M)  and  either  SD  or  ±1  SEM  was  computed  for  measures  across 
the  12  controllers  who  participated.  Some  measures  were  also  summarized  by  controller  position 
(R-side,  D-side)  where  appropriate.  Most  of  the  ground-side  data  were  also  analyzed  using 
ANOVA,  and  the  results  are  reported  in  Appendix  G.  However,  due  to  the  limited  number  of 
observations  in  this  experimental  design,  the  results  should  be  interpreted  with  caution.  This 
study  is  a  preliminary  investigation,  and  the  results  should  not  be  generalized  or  accepted  as 
conclusive. 

3.1.1  Operational  Issues  that  Affect  Shared-Seoaration  Operations 

3. 1 . 1 . 1  Controller  Ratings  for  the  Amount  of  Time  Available  to  Assure  Safe  Aircraft  Separation 
and  Complete  Required  Coordination 

Figure  1 1  shows  mean  controller  ratings  (±1  SEM)  for  time  available  to  assure  safe  aircraft 
separation  and  complete  required  coordination.  In  general,  controllers  rated  the  amount  of  time 
available  for  both  tasks  as  adequate.  Controllers  reported  slightly  more  time  available  to  assure 
safe  aircraft  separation  in  CO;CDTI.  There  were  no  differences  between  the  four  conditions  for 
the  time  available  to  complete  required  coordination.  The  results  of  the  ANOVAs  are  reported  in 
Appendix  G,  Section  G- 1 . 
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Figure  1 1 .  Controller  mean  ratings  for  time  available  for  separation  and  coordination. 


Figure  12  shows  mean  controller  ratings  (±1  SEM)  of  the  level  of  safety  for  procedures 
compared  to  current  operations.  Controllers  rated  the  level  of  safety  for  CO  and  CO:CDTI  as 
unchanged  from  current  operations.  However,  controllers  rated  the  level  of  safety  for  SS:L1  and 
SS:L2  as  compromised.  The  results  of  the  ANOVAs  are  reported  in  Appendix  G,  Section  G-2. 


Compromised  „ 

Level  of  Safety  for  Procedures 

■  CO  BCO:CDTl  BSSiLl  B  SS:L2 
Figure  12.  Controller  mean  ratings  of  level  of  safety  for  procedures. 
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3.1 .1.3  URET  Conflict  Alerts 


Figure  13  shows  mean  frequencies  (number  per  run)  of  URET  red  and  yellow  alerts  (±1  SEM)'^. 
A  URET  alert  for  an  aircraft  pair  was  only  counted  once  and  at  the  highest  level  of  alert.  For 
example,  if  an  aircraft  pair  had  a  yellow  alert  that  progressed  to  a  red  alert  (without  interruption), 
then  the  alert  was  only  counted  once,  as  a  red  alert.  However,  if  an  alert  on  an  aircraft  pair  was 
terminated  and  then  later  reestablished,  it  was  counted  again.  There  were  slightly  more  red  alerts 
than  yellow  alerts  for  all  conditions,  and  only  small  differences  among  the  foiu*  conditions  for 
both  alerts.  The  results  of  the  ANOVAs  are  reported  in  Appendix  G,  Section  G-3A. 
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Figure  13.  Mean  frequency  of  URET  conflict  alerts. 

Figure  14  shows  the  mean  duration  per  alert  (±1  SEM)  for  URET  red  and  yellow  alerts.  In 
general,  the  duration  of  red  and  yellow  alerts  were  longer  in  SS:L2.  The  results  of  the  ANOVAs 
are  reported  in  Appendix  G,  Section  G-3B. 


By  design,  there  were  very  few  ascending  or  descending  aircraft  in  the  scenarios,  therefore,  there  were  very  few 
muted  red  or  yellow  alerts.  In  addition,  these  alerts  were  not  a  primary  focus  of  the  study.  For  these  reasons,  muted 
alerts  were  not  analyzed.  There  were  no  blue  alerts  in  the  study  because  there  was  no  SUA  in  the  sectors  emulated. 
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Figure  14.  Mean  duration  of  URET  red  and  yellow  conflict  alerts. 

3. 1 . 1 .4  Loss  of  Separation  for  Conflicts  Involving  WJHTC  Simulation  Pilots. 

During  the  simulation,  there  were  two  separate  instances  where  aircraft  lost  standard  separation. 
The  two  separation  violations  occurred  during  SS:L2  runs.  From  an  examination  of  the  Host 
system  data  and  videotapes,  both  instances  were  determined  to  be  due  to  unintentional  deviations 
from  the  simulation  pilot  scripts  and  not  the  result  of  shared-separation  operations.  The  video 
tapes  also  confirmed  that  in  both  instances  the  controllers  were  aware  that  the  aircraft  were  about 
to  lose  separation,  but  there  was  nothing  they  could  do  because  they  were  not  allowed  to  cancel 
free  flight  in  the  SS:L2  condition.  One  of  the  instances  involved  a  planned  conflict  where  a 
WJHTC  simulation  pilot  entered  a  command  into  the  system  later  than  the  scripted  time  resulting 
in  a  minimum  horizontal  separation  distance  of  4.87  nm  at  the  same  altitude.  The  second 
instance  involved  an  unplanned  conflict  where  a  WJHTC  simulation  pilot  descended  an  aircraft 
that  was  not  scripted  to  change  altitude  resulting  in  a  minimum  horizontal  separation  distance  of 
4  nm  and  900  ft  altitude  separation. 

3. 1 . 1 .5  MSP  Data  and  Free  Flight  Cancellations  for  Planned  Conflicts  Involving  WJHTC 
Simulation  Pilots 

Table  13  and  Table  14  show  the  number  of  conflicts  and  mean  MSDs  with  SDs  for  altitude- 
resolved  and  vector-resolved  conflicts,  respectively.  MSD  can  be  calculated  by  several  methods. 
One  popular  method  employs  calculating  slant  range  distances.  Another  method,  the  one  used 
for  this  study,  considers  the  perspective  of  the  controller  and  computes  the  measurement  in  terms 
of  the  horizontal  and  vertical  separation  standards  criteria.  Therefore,  MSD  for  an  altitude- 
resolved  conflict  represents  the  minimum  horizontal  distance  between  the  aircraft  pair  imtil 
standard  vertical  separation  was  achieved  (i.e.,  1,000  ft  below  FL290  and  2,000  ft  above  FL290). 
The  MSD  for  a  vector-resolved  conflict  represents  the  minimum  horizontal  distance  between 
aircraft  at  the  same  altitude.  The  MSDs  for  altitude-resolved  conflicts  were  much  greater  for  CO 
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Table  13.  Descriptive  Statistics  for  Altitude-Resolved  Planned  Conflicts  Involving  WJHTC 

Simulation  Pilots 


Control 

Condition 

Controller  Altitude-Resolved 
Conflicts 

Scripted  Pilot  Altitude-Resolved 
Conflicts 

N 

Mean  MSD 

SD 

N 

Mean  MSD 

SD 

CO 

16 

32.5  nm 

19.7  nm 

Pilots  could  not  initiate  maneuvers  in  this 
condition 

CO:CDTI 

17 

45.8  nm 

19.3  nm 

Pilots  could  not  initiate  maneuvers  in  this 

_ rnnditinn _ _ _ 

SS:L1 

3* 

14.8  nm 

0.3  nm 

15 

13.4  nm 

4.6  nm 

SS:L2 

Controllers  could  not  cancel  free  flight  in  this 
condition 

3 

7.4  nm 

2.7  nm 

Note:  The  MSD  for  an  altitude-resolved  conflict  represents  the  minimum  horizontal  distance 
between  the  aircraft  pair  until  standard  vertical  separation  was  achieved  (i.e.,  1,000  ft  below 
FL290  and  2,000  ft  above  FL290). 

*  Conflicts  with  cancelled  free  flight.  _ 


Table  14.  Descriptive  Statistics  for  Controller  Vector-Resolved  Planned  Conflicts  Involving 

WJHTC  Simulation  Pilots 


Control 

Condition 

Controller  Vector-Resolved 
Conflicts 

Scripted  Pilot  Vector-Resolved 
Conflicts 

N 

Mean  MSD 

SD 

N 

Mean  MSD 

SD 

CO 

22 

17.3  nm 

19.00  nm 

Pilots  could  not  initiate  maneuvers  in  this 
condition 

CO:CDTI 

20 

15.7  nm 

10.16  nm 

Pilots  could  not  initiate  maneuvers  in  this 
condition 

9* 

12.4  nm 

5.30  nm 

12 

13.1  nm 

7.3  nm 

SS:L2 

Controllers  could  not  cancel  free  flight  in  this 
condition 

36 

10.0  nm 

7.2  run 

Note:  The  MSD  for  a  vector-resolved  conflict  represents  the  minimum  horizontal  distance 
between  aircraft  at  the  same  altitude. 


*Conflicts  with  cancelled  free  flight. 
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and  CO:CDTI  compared  to  SS;L1  and  SS:L2.  Considering  the  means  and  SDs  of  MSDs  for 
vector-resolved  conflicts,  it  was  not  possible  to  establish  if  a  difference  exists  between 
conditions.  In  SS:L1,  controllers  cancelled  free  flight  and  resolved  the  conflicts  for  12  conflict 
pairs  (31%)  and  allowed  WJHTC  simulation  pilots  to  resolve  the  conflicts  for  27  pairs  (69%). 
Controllers  used  altitude  to  resolve  three  of  the  conflict  pairs  and  used  vectors  to  resolve  nine  of 
the  pairs  when  free  flight  was  cancelled.  The  results  of  the  ANOVAs  for  MSDs  are  reported  in 
Appendix  G,  Section  G-4. 

3.1.2  Information  Requirements  and  Procedures 

3. 1 .2. 1  Controller  Ratinus  for  the  Amount  of  Information  Available  to  Resolve  Conflicts 

Figure  15  shows  mean  controller  ratings  (il  SEM)  for  information  available  to  resolve  conflicts. 
In  general,  controllers  rated  the  amount  of  information  to  resolve  conflicts  as  adequate,  and  there 
were  no  differences  between  the  four  conditions.  The  results  of  the  ANOVA  are  reported  in 
Appendix  G,  Section  G-5. 
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Information  to  Resolve  Conflicts 
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Figure  15.  Controller  mean  ratings  of  information  to  resolve  conflicts. 

3. 1.2.2  Controller  Ratines  for  URET  Conflict  Alert  Timeliness 

Figure  16  shows  mean  controller  ratings  (±1  SEM)  for  URET  conflict  alert  timeliness.  In 
general,  controllers  rated  the  timing  of  the  conflict  alerts  as  adequate  and  there  were  no 
differences  between  the  four  conditions.  The  results  of  the  ANOVA  are  reported  in  Appendix  G, 
Section  G-6. 
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Figure  16.  Controller  mean  ratings  for  URET  conflict  alert  timeliness. 

3.1 .2.3  Controller  Ratings  for  How  Often  They  Monitored  Air<-»Air  Communications 

Table  15  shows  the  frequencies  and  percentages  of  controller  responses  for  ratings  of  how  often 
they  monitored  air<->air  communications.  The  majority  of  controllers  reported  that  they  always 
monitored  air<^air  communications,  and  there  was  no  difference  between  SS:L1  and  SS;L2.  In 
both  SS:L1  and  SS:L2,  three  of  the  six  (50%)  R-side  controllers  always  monitored  and  three 
(50%)  never  monitored  air<->air  communications.  In  both  SS:L1  and  SS:L2,  all  six  (100%)  of 
the  D-side  controllers  always  monitored  airoair  communications.  The  researchers  did  not 
perform  an  ANOVA  on  controller  ratings  for  how  often  they  monitored  air<^air  communications 
due  to  lack  of  variability  in  the  data. 


Table  15.  Controller  Frequencies  of  Air^^-^Air  Communication  Monitoring 


Control 

Condition 

R-Side  Controllers 

D-Side  Controllers 

1 

Never 

2 

— 

3 

Some 

4  5 

1  Always 

1 

Never 

2 

3 

Some 

4 

5 

Always 

SS:L1 

0 

SS:L2 

3  (50%) 

- ' 

0 

- j 

0 

0  1  3(50%) 

1 

0 

0 

0 

0 

6(100%) 
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3.1 .2.4  Controller  Ratines  for  the  Usefiilness  of  Monitoring  Air<-»Air  Communications 


Figure  17  shows  mean  controller  ratings  (+1  SEM)  for  the  usefulness  of  monitoring  air'<->air 
communications.  Only  the  ratings  from  controllers  who  monitored  air4->air  communications  are 
represented  in  this  figure.  In  general,  controllers  rated  monitoring  airoair  communications  as 
useful,  and  there  was  no  significant  difference  between  SS;L1  and  SS:L2.  In  addition,  there  was 
no  significant  difference  between  R-side  and  D-side  controllers.  The  results  of  the  ANOVA  are 
reported  in  Appendix  G,  Section  G-7. 
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Figure  1 7.  Controller  mean  ratings  for  usefulness  of  airoair  communications. 

3.1 .2.5  Controller  Ratines  for  the  Helpfulness  of  the  Shared-Separation  Concept 

Figure  18  shows  mean  controller  ratings  (±1  SEM )  for  the  helpfulness  of  the  shared-separation 
concept.  In  general,  controllers  rated  the  shared-separation  concept  as  not  helpful,  and  there  was 
no  significant  difference  between  SS:L1  and  SS:L2.  The  results  of  the  ANOVA  are  reported  in 
Appendix  G,  Section  G-8. 
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Figure  18.  Controller  mean  ratings  for  the  helpfulness  of  shared-separation. 

3. 1.2.6  URET  Trial  Plans 

Figure  19  shows  mean  frequencies  (±1  SEM)  of  URET  trial  plans.  In  general,  controllers 
formed  more  trial  plans  in  CO  and  CO:CDTI  compared  to  SS;L1  and  SS:L2.  The  results  of  the 
ANOVA  are  reported  in  Appendix  G,  Section  G-9. 
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Figure  19.  Mean  frequency  of  URET  trial  plans. 


48 


3.1.3  Controller  Workload  and  Situation  Awareness 


3. 1 .3. 1  Controller  Ratings  for  Physical.  Mental,  and  Overall  Workload 

Figure  20  shows  mean  controller  ratings  (±1  SEM)  for  physical,  mental,  and  overall  workload. 
In  general,  controllers  rated  their  workload  as  moderate.  Mental  and  overall  workload  ratings 
were  higher  for  SS:L1  compared  to  the  other  conditions.  The  results  of  the  ANOVAs  are 
reported  in  Appendix  G,  Section  G-10. 
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Figure  20.  Controller  mean  workload  ratings:  physical,  mental,  and  overall. 

3. 1.3. 2  Controller  Workload  Ratines  for  Maintaining  Aircraft  Separation.  Land  Line 
Coordination.  R-Side-to-D-Side  Coordination.  Ground->Air  Transmissions,  and  URET 
Coordination 


Figure  21  shows  mean  controller  workload  ratings  (±1  SEM)  for  maintaining  aircraft  separation, 
land  line  coordination,  R-side-to-D-side  position  coordination,  ground->air  transmissions,  and 
URET  coordination.  In  general,  controllers  rated  their  workload  for  these  specific  areas  as 
ranging  from  low  to  moderate.  Controller  workload  was  generally  the  highest  for  maintaining 
aircraft  separation  and  the  lowest  for  URET  coordination.  There  were  no  significant  differences 
in  workload  among  the  four  conditions  for  these  specific  measures.  The  results  of  the  ANOVAs 
are  reported  in  Appendix  G,  Section  G-1 1. 
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Figure  21.  Controller  mean  workload  ratings:  specific  measures. 

3. 1.3. 3  Controller  Ratines  for  Feeling  Rushed  and  Bored 

Figure  22  shows  mean  controller  ratings  (±1  SEM)  for  feeling  rushed  and  bored  during  the 
simulation.  In  genereil,  controllers  felt  neither  rushed  nor  bored,  and  there  were  no  significant 
differences  between  the  four  conditions.  The  results  of  the  ANOVAs  are  reported  in 
Appendix  G,  Section  G-12. 
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Figure  22.  Controller  mean  ratings  for  feeling  rushed  and  bored. 
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3. 1.3. 4  Controller  Ratings  of  Overall  Situation  Awareness 


Figure  23  shows  mean  controller  ratings  (±1  SEM)  of  overall  situation  awareness.  In  general, 
controllers  rated  their  overall  situation  awareness  as  high  and  there  were  no  significant 
differences  between  the  four  conditions.  The  results  of  the  ANOVA  are  reported  in  Appendix  G, 
Section  G-13. 


Figure  23.  Controller  mean  ratings  of  overall  situation  awareness. 

3. 1.3. 5  Controller  Interval  Workload  Ratines 

Figure  24  and  Figure  25  show  mean  interval  workload  ratings  for  the  R-side  and  D-side 
controllers,  respectively.  In  general,  controllers  rated  their  workload  using  the  WAK  tool  as 
rather  low.  As  shown  in  the  figures,  lower  and  higher  workload  periods  can  be  seen  for  each  of 
the  four  conditions.  Each  condition  had  two  peak  workload  periods:  the  first  occurred  at 
approximately  25-35  minutes  into  the  run,  and  the  second  was  at  about  60-70  minutes  into  the 
run.  In  contrast  to  the  results  for  the  controller  ratings  of  overall  workload  (section  5. 1 .3. 1), 
there  were  only  small  differences  between  the  four  conditions  for  interval  workload  ratings.  The 
results  of  the  ANOVA  for  mean  workload  ratings  collapsed  across  the  intervals  are  reported  in 
Appendix  G,  Section  G-14. 
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Figure  24.  R-side  controller  mean  ratings  for  interval  workload. 
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Figure  25.  D-side  controller  mean  ratings  for  interval  workload. 

3. 1.3.6  Expert  Observer  Ratines  of  Controller  Physical  Taskload 

Figure  26  shows  mean  EO  ratings  (±1  SEM)  of  controller  physical  taskload.  In  general,  EOs 
rated  controller  physical  taskload  as  moderate.  Similar  to  the  results  of  controller  ratings  for 
overall  workload,  EO  ratings  for  controller  physical  taskload  were  higher  in  SS:L1  compared  to 
the  other  conditions.  The  results  of  the  ANOVA  are  reported  in  Appendix  G,  Section  G-15. 
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Figure  26.  Expert  Observer  mean  ratings  of  controller  physical  taskload. 

3.1 .3.7  Controller  Ground-^ Air  and  Land  Line  Push-to-Talk  Transmissions 

Figure  27  shows  mean  frequencies  (±1  SEM)  of  ground-^air  and  land  line  PTTs.  There  were 
many  more  ground^air  PTTs  than  land  line  PTTs,  and  both  types  were  the  lowest  in  SS:L2.  It 
should  be  noted  that  by  design,  SS:L2  had  no  scripted  pilot  inquires,  requests,  or  intent 
information  relays  (and,  therefore  no  controller  responses  or  acknowledgments).  In  addition, 
controllers  could  not  cancel  free  or  issue  control  instructions  to  pilots.  These  factors  likely 
contributed  to  the  decrease  in  SS:L2  PTTs.  The  results  of  the  ANOVAs  are  reported  in 
Appendix  G,  Section  G-16A. 
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Figure  27.  Mean  frequency  of  ground->air  and  land  line  PTTs. 
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Figure  28  shows  mean  duration  per  transmission  (±1  SEM)  of  ground->air  and  land  line  PTTs. 
Ground->air  PTTs  were  shorter  than  land  line  PTTs,  and  there  were  only  small  differences 
between  the  four  conditions.  The  results  of  the  ANOVAs  are  reported  in  Appendix  G, 

Section  G-16B. 


Figure  28.  Mean  duration  per  transmission  of  ground-^air  and  land  line  PTTs. 

3.1.4  Exit  Form  Responses  and  Ratines 

Tables  16  and  17  show  controller  responses  about  separation  responsibility  confusion  and  ratings 
of  simulation  realism  and  training  obtained  from  the  Exit  Form.  Nine  controllers  reported  that 
they  were  not  confused  about  who  had  the  separation  responsibility.  However,  three  controllers 
reported  that  they  were  confused  at  some  time  but  did  not  indicate  how  long.  In  general, 
controllers  rated  the  realism  of  the  simulation  pilot  responses  and  overall  realism  of  the 
simulation  as  moderate.  In  addition,  controllers  rated  the  simulation  training  as  moderate  to 
adequate. 


Table  16.  Controller  Role  and  Separation  Responsibility  Confusion 


N  =  12 

YES 

NO 

Confused  at  any  time  about 
separation  responsibility  and  role 

n  =  3 

(R-side  =  2  &  D-side  =1) 
(25%) 

n  =  9 

(R-side  =  4  &  D-side  =5) 
(75%) 
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Table  17.  Controller  Mean  Ratings  for  Simulation  Realism  and  Training 


N  =  12 

M  SD 

Realism  of  the  simulated  flight  crew  responses 
(1  =  very  unrealistic,  3  =  moderate,  5  =  very  realistic) 

3.3 

0.9 

Overall  realism  of  the  simulation 
(1  =  very  unrealistic,  3  =  moderate,  5  =  very  realistic) 

3.0 

0.9 

Adequacy  of  simulation  training 
(1  =  inadequate,  3  =  moderate,  5  -  adequate 

3.9 

1.1 

3.2  Air-Side  Results 

Due  to  the  limited  number  of  participants  in  this  study  (N  =  3  flight  crews  for  a  total  of  6  pilot 
participants),  no  inferential  statistics  were  applied  to  the  air-side  data.  Simulation  pilots 
performing  scripted  events  operated  the  other  aircraft  in  this  investigation;  therefore,  those  data 
will  not  be  discussed  in  this  section. 

Descriptive  statistics  are  provided  for  the  various  measures  collected  during  the  study.  In 
addition.  Pilot  Exit  Form  data  were  analyzed  using  the  Analytical  Hierarchy  Process  (AHP). 

AHP  compared  ratings  of  measures  across  conditions  on  a  mirrored  9-point  scale.  AHP  factor 
loadings  were  compared  using  a  standard  “level  of  dominance”  scale  where  (1  =  equal, 

3  =  weak,  5  =  strong,  7  =  very  strong,  and  >9  =  an  absolute  preference).  These  ratings 
indicated  the  dominance  of  one  condition  over  another  in  terms  of  pilot  preference.  Therefore,  a 
factor  of  1  would  indicate  that  both  of  the  conditions  being  compared  were  equally  preferred  (or 
no  preference),  and  a  factor  of  9  would  indicate  an  absolute  (or  overwhelming)  preference  for 
one  of  the  conditions  over  the  other. 

Except  where  noted,  statistics  summarize  across  the  three,  20-minute  flight  segments  in  each  of 
the  four  conditions.  Thus,  there  were  36  flight  segments  considered  for  most  of  these  data 
summaries.  These  data  must  be  interpreted  with  caution  because  of  the  small  number  of 
participants. 

3.2.1  Operational  Issues  that  Affect  Shared-Senaration  Operations 

3.2. 1.1  Safety  Measures 

•  Loss  of  Separation  for  Conflicts  Involving  the  NASA  ARC  Simulator. 

In  the  two  shared-separation  conditions  (SS:L1  and  SS:L2),  the  flight  crews  had  varying 
levels  of  separation  responsibility.  For  all  conditions,  the  minimum  separation  distance  was 
defined  as  either  5  nm  horizontal  or  1000/2000  ft  vertical  (as  appropriate).  There  were  no 
NASA  flight  segments  in  which  minimum  separation  standards  were  violated  by  the  flight 
crews  in  the  four  conditions  (CO,  CO:CDTI,  SS:L1,  SS;L2).  In  addition,  there  were  no  flight 
segments  in  which  the  flight  crews  received  any  TCAS  alerts. 
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•  AHP  Flight  Safety  Ratings. 

Pilot  participants  were  asked  which  condition  they  preferred  in  terms  of  flight  safety.  Table 
18  shows  the  AHP  results  of  these  data.  Interestingly,  AHP  preference  ratings  indicated  that 
pilots  preferred  the  conditions  in  which  they  assumed  some  responsibility  for  separation  over 
the  more  traditional  roles  (SS:L2  to  CO  by  a  factor  of  9.5  and  SS:L2  over  SS:L1  by  a  factor 
of  2.9).  The  least  favored  conditions  were  CO  and  CO:CDTI. 


Table  18.  Pilot  Participant  AHP  Preference  Ratings  for  Flight  Safety 


Flight  Safety  Comparison 

Factor 

Level  of  dominance 

CO:CDTI  equal  to  CO 

1.4 

equal 

SS:L1  preferred  over  CO 

3.3 

weak 

SS:L2  preferred  over  CO 

9.5 

absolute 

SS:L1  preferred  over  CO;CDTI 

2.3 

weak 

SS:L2  preferred  over  COrCDTI 

6.7 

very  strong 

SS:L2  preferred  over  SS:L1 

2.9 

weak 

3 .2. 1 .2  Flight  Crew  Performance 
•  Flight  Crew  Conflict  Detection. 

Flight  crews  could  identify  potential  conflicting  aircraft  using  the  information  provided  by 
the  CDTl-AL.  For  each  of  the  27  flight  segments  in  which  this  information  was  available 
(those  from  CO;CDTI,  SS:L1,  and  SS:L2),  video  and  audio  tapes  from  the  flight  deck  were 
coded  by  two  researchers  to  determine  the  time  at  which  the  intruder  aircraft  was  identified 
as  a  possible  conflict  by  the  flight  crews.  Although  the  flight  crews  were  not  given 
separation  flexibility  in  CO:CDTl,  they  did  appear  to  monitor  their  CDTI-AL  in  that 
condition.  Conflict  detection  time  was  defined  as  duration  from  the  time  the  intruder  aircraft 
appeared  on  the  CDTI-AL  (at  the  beginning  of  each  flight  segment)  until  both  members  of 
the  flight  crew  indicated  that  the  aircraft  was  a  potential  problem  (also  determined  by  video 
transcription  analysis).  In  26  out  of  the  27  flight  segments  with  CDTI-AL  information,  the 
intruder  aircraft  was  identified  by  the  flight  crew  as  a  potential  conflict.  In  the  remaining 
run,  the  flight  crew  made  a  horizontal  maneuver  to  go  direct  to  a  subsequent  fix  along  their 
route.  It  was  unclear  whether  the  maneuver  was  made  because  of  the  potential  for  conflict 
with  the  intruder  aircraft,  for  fuel  efficiency,  or  for  both.  This  maneuver  resolved  the 
conflict,  so  no  conflict  detection  time  was  generated.  Table  19  shows  mean  conflict 
detection  times  and  SDs.  These  average  times  were  at  least  4  minutes  before  the  time  that  the 
intruder  aircraft  triggered  an  alert  on  the  CDTI-AL.  The  pilot  participants  detected  all 
conflicts  prior  to  a  CDTI-AL  alert  indication.  It  should  be  noted  that  the  conflicts  all  had  the 


57 


same  general  angle  and  location,  and  the  pilot  participants  did  indicate  that  they  identified 
this  pattern  during  the  experiment. 

Table  19.  Pilot  Participant  Mean  Intruder-Detection  Times 


CO;CDTI 

SS:L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

M 

SD 

Pilot  Intruder  Aircraft 

Detection  Times  (min:sec) 

2:17 

1:30 

1:48 

1:41 

1:34 

1:28 

In  addition  to  determining  how  long  flight  crews  took  to  identify  the  conflicting  aircraft, 
following  each  condition  they  were  asked  to  rate  1)  the  ease  of  detecting  a  conflict  prior  to  an 
alert  indication  on  the  CDTI-AL  or  controller  advisory,  and  2)  the  effectiveness  of  the  CDTI-AL 
for  use  in  a  shared-separation  environment.  The  means  and  SDs  are  presented  in  Tables  20  and 
21 .  Similar  to  the  flight  crew  conflict  detection  times,  inspection  of  the  subjective  ratings  data 
suggest  that  flight  crews  found  it  easy  to  identify  the  intruder  aircraft  in  all  three  of  the 
conditions  that  included  the  CDTI-AL,  and  that  the  CDTI-AL  was  an  effective  tool  in  identifying 
the  conflicting  aircraft. 

Table  20.  Pilot  Participant  Mean  Ratings  of  the  Ease  of  Detecting  Conflicts  Prior  to  Alert  or 

Controller  Advisory 


COrCDTl 

SS:L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

M 

SD 

Ease  of  detecting  conflicts  prior 
to  alert  or  controller  advisory 

3.7 

1.5 

3.7 

1.5 

4.7 

0.5 

(I  =  not  easy,  3  =  moderately  easy,  5  =  very  easy) 
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Table  21.  Pilot  Participant  Mean  Ratings  of  the  CDTl-AL  Effectiveness  for  Shared-Separation 


SS:L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

Effectiveness  of  display  for 
shared-separation 

4.2 

1.0 

4.3 

0.8 

(/  =  not  effective,  3  =  moderately  effective,  5  =  very  effective) 


3 .2  ■  1 .3  Flight  Efficiency  Measures 

•  Fuel  bum. 

Because  of  the  different  routes  through  the  sectors  and  due  to  the  short  flight  segments,  fuel 
bum  comparisons  could  not  be  made  among  the  conditions.  Futme  studies  need  to  consider 
longer  flight  segments  in  order  to  estimate  potential  impacts  of  fuel  bum  in  various  shared- 
separation  conditions. 

•  Pilot  participant  ratings  of  flight  efficiency. 

After  flying  all  of  the  conditions,  flight  crews  were  asked  to  rate  the  flight  efficiency  of  each 
procedural  condition  using  the  AHP.  Pilot  participants  were  asked  which  authority 
conditions  they  considered  better  for  flight  efficiency.  Again,  as  can  be  seen  in  Table  22,  the 
pilots  overwhelmingly  preferred  SS:L2  over  CO  and  COrCDTI.  They  preferred  SS:L2  over 
CO  by  a  factor  of  12.2,  over  CO:CDTI  by  a  factor  of  10.1  and  over  SS:L1  by  a  factor  of  3.2. 
SS:L1  was  preferred  over  CO  and  CO:CDTl  by  factors  of  3.8  and  3.1,  respectively. 


Table  22.  Pilot  Participant  Preference  for  Flight-Efficiency 


Flight  Efficiency  Comparison 

Factor 

Level  of  dominance 

CO:CDTI  equal  to  CO 

1.2 

equal 

SS:L1  preferred  over  CO 

3.8 

weak 

SS:L2  preferred  over  CO 

12.2 

absolute 

SS:L1  preferred  over  CO:CDTI 

3.1 

weak 

SS:L2  preferred  over  CO:CDTl 

10.1 

absolute 

SS:L2  preferred  over  SS:L1 

3.2 

weak 
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3.2. 1.4  Flight  Crew  Communication 
•  Air<->Ground  Communication. 


After  flying  in  each  condition,  pilot  participants  were  asked  to  rate  how  much  time  they  felt 
was  available  for  air4^ground  communications.  Table  23  depicts  their  mean  ratings  and  SDs 
for  each  condition.  Across  all  conditions,  pilots  reported  an  adequate  amount  of  time 
available  for  air<^ground  communications. 


Table  23.  Pilot  Participant  Mean  Ratings  for  Time  Available  for  Air<->Ground  Communication 


CO 

CO:CDTl 

SS:L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

Amount  of  time 
available  for 
air4->ground 
communication 

3.3 

1.0 

3.2 

0.4 

3.0 

.0 

3.2 

0.4 

(/  =  too  little,  3  =  adequate,  5  =  too  much) 


•  Air<-»Air  Communication. 

In  SS:L1  and  SS:L2,  an  air<-)^air  frequency  was  available  on  which  all  pilots  (pilot 
participants  and  simulation  pilots)  were  able  to  communicate  among  themselves.  Because 
the  simulation  pilots  were  confederates  of  the  study,  their  communication  duration  times 
were  not  calculated.  Simulation  pilots  did  not  initiate  communication  with  the  pilot 
participants  (scripted  or  otherwise).  Pilot  participants  contacted  the  intruder  aircraft  in  only 
three  (33%)  of  the  nine  flight  segments  in  SS:L1  and  in  four  (44%)  of  the  nine  flight 
segments  in  SS:L2.  Pilot  participants  contacted  one  other  aircraft  in  SS;L1  and  none  of  the 
surrounding  aircraft  other  than  the  intruder  in  SS:L2. 

Following  each  condition,  pilot  participants  were  asked  to  rate  several  questions  concerning 
air<->^air  communication;  how  often  they  monitored  the  air<->air  frequency,  the  usefulness  of 
airo^air  communication,  and  the  amount  of  time  available  for  monitoring  air4->air 
communication.  Table  24  summarizes  pilot  participant  responses  for  all  questions.  In  SS:L1, 
pilots  reported  that  they  frequently  monitored  the  communications  between  the  other  pilots  and 
they  sometimes  monitored  it  in  SS:L2.  Pilots  reported  that,  on  average,  monitoring  the  air<->air 
fi'equency  was  moderately  useful,  and  they  had  an  adequate  amount  of  time  to  monitor  the 
air<->air  frequency  in  both  shared-separation  conditions. 
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Table  24.  Pilot  Participant  Mean  Ratings  Related  to  Air<^Air  Communication 


SS;L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

How  often  monitored  other  air<->air  communication 

(7  =  never,  3  =  sometimes,  5  =  always) 

4.0 

0.9 

3.3 

0.8 

Usefulness  of  monitoring  other  air<->air  communication 

(1  =  not  useful,  3  =  moderately  useful,  5  =  very  useful) 

3.0 

1.1 

B 

0.5 

Amount  of  time  available  to  monitor  air<->air  frequency 

(1  =  too  little,  3  =  adequate,  5  =  too  much) 

2.8 

0.4 

3.3 

0.5 

3.2.2  Information  Requirements  and  Procedures 
3.2.2. 1  Procedures 

During  the  study,  flight  crews  flew  three  conditions  in  which  different  procedures  and/or  new 
technologies  were  provided  as  well  as  a  fourth  condition  using  current  operational  procedures. 
Flight  crews  were  briefed  on  both  the  procedures  for  themselves  as  pilots  and  on  the  new 
procedures  for  the  controllers.  After  flying  in  each  condition,  flight  crews  were  queried  about 
the  procedures  used  for  shared-separation  in  this  study;  means  and  SDs  for  their  form  responses 
are  presented  in  Table  25.  According  to  these  responses,  flight  crews  felt  that  the  operations 
used  in  both  SS:L1  and  SS:L2  were  helpful  for  performing  their  jobs,  and  they  felt  comfortable 
sharing  the  separation  responsibility  with  controllers.  Pilot  participants  also  reported  only  slight 
confusion  about  who  had  separation  authority  during  the  runs.  Table  26  shows  the  pilots  mean 
ratings  and  SDs.  This  confusion  was  also  noticed  in  a  few  comments  by  the  pilots  (found  during 
videotape  analyses).  Finally,  pilot  participants  indicated  that  the  pilot  right-of-way  rules  were 
useful  for  negotiation  and  initiating  contact  with  conflicting  aircraft  (M  =  4.0, 5D  =  1.1;  /  =  not 
useful,  3  =  moderately  useful,  5  =  very  useful). 
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Table  25.  Pilot  Participant  Mean  Ratings  of  Shared-Separation 


SS:L1 

SS:L2 

N=6 

M 

SD 

M 

SD 

Impact  of  shared-separation  operations  on  performing  job. 

(1  =  detrimental,  3  =  no  impact,  5  =  helpful) 

4.0 

1.6 

■ 

0.8 

Comfort  in  sharing  separation  responsibility. 

(1  -  not  comfortable,  3  =  moderately  comfortable, 

5  =  very  comfortable) 

4.2 

1.2 

1 

0.8 

Table  26.  Pilot  Participant  Mean  Ratings  of  Separation  Responsibility  Confusion 


CO 

CO:CDTI 

SS:L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

M 

SD 

M 

SD 

Uncertainty  of  who 
had  separation 
authority. 

1.2 

0.4 

1.0 

0 

1.8 

1.0 

1.0 

0 

(1  =  very  low,  3  =  moderate,  5  =  very  high) 


3.2.22  Pilot  Information  Requirements 

Following  the  simulation,  pilot  participants  were  asked  several  questions  regarding  the  use  and 
the  appropriateness  of  the  tools  provided.  Although  the  pilots  usually  detected  the  conflicts  prior 
to  the  alerting,  the  data  revealed  that  they  found  the  CDTI  and  the  associated  conflict  alerting 
logic  quite  effective  for  safe  operations.  The  pilot  participants  also  indicated  that  the  amount  of 
information  on  the  CDTI-AL  was  adequate  to  identify  and  resolve  conflicts  and  that  the  timing 
of  the  conflict  alert  was  adequate  for  strategic  separation  tasks.  Table  27  summarizes  the  pilots 
responses.  Data  from  the  Post-Run  Forms  found  that  pilot  participants  felt  the  timeliness  of  the 
CDTI-AL  conflict  alert  was  adequate  across  the  conditions.  Mean  ratings  and  SDs  are  reported 
in  Table  28. 
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Table  27.  Pilot  Participant  Mean  Ratings  of  Information. 


N  =  6 

M  SD 

Effectiveness  of  CDTI  and  alerting  for  safe  operations. 

(1  =  not  effective,  3  =  moderately  effective,  5  =  very  effective) 

4.8 

0.4 

Amount  of  information  on  CDTI  to  identify  and  resolve  conflicts. 

(1  =  too  little,  3  =  adequate,  5  =  too  much) 

3.5 

0.8 

Time  CDTI  alerting  provided  for  strategic  shared-separation. 

(1  =  too  little,  3  =  adequate,  5  =  too  much) 

3.3 

0.5 

Table  28.  Pilot  Participant  Mean  Ratings  of  the  Timeliness  of  CDTI-AL  Alerts 


COrCDTI 

SS:L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

M 

SD 

Timeliness  of  CDTI-AL  alert 

3.3 

0.5 

3.0 

0 

3.0 

0 

(1  =  too  early,  3  =  adequate,  5  =  too  late) 


Pilot  form  data  indicated  that  flight  crews  spent  a  considerable  amount  of  time  monitoring  the 
CDTI-AL  with  the  average  percentage  of  time  increasing  as  the  pilot  responsibility  for 
separation  increased.  Table  29  displays  means  and  SDs  for  monitoring  time.  Flight  crews 
reported  spending  about  two-thirds  of  the  time  monitoring  the  CDTI-AL,  therefore,  it  is 
important  to  note  that  in  addition  to  the  CDTI-AL  information,  the  Navigation  Display  served  as 
an  important  data  source  for  information  on  their  current  and  programmed  route  of  flight, 
weather,  and  other  FMS  information.  However,  concerns  about  spending  too  much  time 
monitoring  the  CDTI  were  mentioned  consistently  by  nearly  all  pilots  involved  in  this  study. 

Table  29.  Mean  Percentage  of  Time  Pilot  Participants  Spent  Monitoring  the  CDTI-AL 


COrCDTI 

SSrLl 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

M 

SD 

Time  spent  monitoring 
CDTI-AL 

62.5% 

20.9% 

66.7% 

21.4% 

68.3% 

21.1% 
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In  order  to  investigate  how  the  flight  crews  may  have  been  using  the  time  they  spent  monitoring 
the  CDTI-AL,  the  researchers  calculated  how  often  flight  crews  used  the  various  CDTI-AL 
functions  provided  in  this  study,  including  the  temporal  predictors,  callsign/ground  speed 
information  and  selectable  map  range  levels.  Of  the  27  flight  segments  in  which  the  CDTI-AL 
was  available  to  the  pilot  participants,  on  average  they  had  the  temporal  predictors  selected 
“ON”  during  89.7%  (SD  =  14.4%)  of  the  flight  segment  time.  In  addition,  call  sign/ground 
speed  information  was  selected  “ON”  for  an  average  of  13.4%  (SD  =  14.6%)  of  the  flight 
segment  time. 

Flight  crews  could  also  use  the  selectable  map  range  functionality  of  the  Navigation  Display  to 
help  de-clutter  their  display.  There  were  seven  hard-coded  map  range  values  that  the  pilots  could 
select  from  on  the  Boeing  747-400  Navigation  Display  (10, 20, 40,  80, 160,  320,  and  640  nm). 
Figure  29  depicts  the  average  amount  of  flight  segment  time  spent  at  each  available  Navigation 
Display  range.  Flight  crews  spent  the  majority  of  the  time  at  the  160  nm  range  (around  55%  of 
the  time  across  conditions),  followed  by  the  80  nm  (roughly  35%  of  the  time).  This  is  consistent 
with  the  proposed  ADS-B  range  of  120  nm,  as  160  nm  selection  provides  the  full  extent  of  that 
range  (RTCA,  1992). 


lOnm  20nm  40nni  SOnin  160nni  320nin  640nni 

Map  Ranges 

S  COCDTl  ■  SS:L1  E  SS:L2  | 

Figure  29.  Mean  percentage  of  time  pilot  participants  spent  at  each  map  range  level. 
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3.2.3  Pilot  Workload  and  Situation  Awareness 
3.2.3. 1  Pilot  Workload 
•  AHP  Workload  Ratings. 

Pilot  participants  were  asked  to  indicate  which  condition  they  considered  better  for  reducing 
workload.  Again  they  responded  with  a  preference  for  SS:L2  over  the  other  conditions 
Table  30  presents  AHP  factor  loadings  and  level  of  dominance  ratings.  Pilots  preferred 
SS:L2  over  CO  by  a  factor  of  5.9,  over  CO:CDTI  by  a  factor  of  2.4  and  over  SS:L1  by  a 
factor  of  2.9.  SS:L1  was  preferred  over  CO  by  a  factor  of  2.7  but  was  equally  preferred  to 
CO:CDTI. 


Table  30.  Pilot  Participant  Preference  for  Reducing  Workload 


Reducing  Workload  Comparison 

Factor 

Level  of  Dominance 

2.5 

weak 

SS:L1  preferred  over  CO 

2.7 

weak 

SS:L2  preferred  over  CO 

5.9 

strong 

SS:L1  equal  to  CO:CDTI 

1.1 

equal 

SS:L2  preferred  over  COrCDTI 

2.4 

weak 

SS:L2  preferred  over  SS;L1 

2.9 

weak 

•  Subjective  Workload  Ratings 

Pilot  participants  were  asked  to  rate  workload  levels  after  the  completion  of  each  condition. 
Results  are  presented  in  Table  31.  In  general,  pilot  workload  ratings  were  low,  only  ranging 
from  very  low  to  low.  Regardless,  inspection  of  the  means  indicates  that  the  ratings  are 
generally  higher  for  SS:L1  when  compared  to  the  other  conditions. 
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Table  31.  Pilot  Participant  Mean  Workload  Ratings 


N  =  6 

CO 

CO:CDTl 

SS:L1 

SS:L2 

M  SD 

M  SD 

M  SD 

M  SD 

Physical 

1.0 

0 

1.0 

0 

1.8 

0.8 

1.5 

0.6 

Mental 

1.0 

0.5 

2.0 

0.6 

1.8 

0.8 

Overall 

1.0 

0.5 

1.8 

0.4 

1.5 

0.6 

Air<^Ground  Comm 

1.2 

0.4 

1.2 

0.4 

1.2 

0.4 

1.3 

0.5 

Crew  Coordination 

1.2 

0.4 

1.0 

0 

1.5 

0.6 

1.2 

0.4 

Aircraft  Separation 

N/A 

N/A 

N/A 

N/A 

2.0 

0.6 

1.8 

0.8 

AiroAir  Comm. 

N/A 

N/A 

N/A 

N/A 

1.5 

0.8 

1.3 

0.5 

(1  =  very  low,  3  =  moderate,  5  =  very  high) 


•  Pilot  Participant  Situation  Awareness  Measures 

Overall  situation  awareness  was  defined  for  the  flight  crews  as  “What  is  commonly  known  as 
the  pilot’s  ‘staying  ahead  of  the  aircraft’  where  the  pilot  has  a  thorough  understanding  of  the 
current  situation  and  can  take  appropriate  action  as  necessary.” 

•  AHP:  Maintaining  Situation  Awareness. 

Using  the  AHP,  pilot  participants  rated  which  condition  they  considered  better  for 
maintaining  situation  awareness.  Table  32  illustrates  the  flight  crew’s  condition  preferences. 
Pilots  favored  SS:L2  for  maintaining  situation  awareness  over  the  other  conditions.  They 
preferred  SS:L2  over  CO  for  maintaining  situation  awareness  by  a  factor  of  9.9,  over 
CO:CDTI  by  a  factor  of  3.3  and  over  SS:L1  by  a  factor  of  2.1.  SS:L1  was  preferred  over  CO 
by  a  factor  of  4.7  but  was  rated  equal  to  CO:CDTl  for  this  question.  Predictably,  pilots 
reported  that  having  a  CDTI-AL  (COrCDTI)  may  have  helped  with  situation  awareness  as 
COiCDTI  was  preferred  over  CO  by  a  factor  of  3.0. 
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Table  32.  Pilot  Participant  Preference  for  Maintaining  Situation  Awareness 


Maintaining  Situation  Awareness 
Comparison 

Factor 

Level  of  dominance 

CO;CDTI  preferred  over  CO 

3.0 

weak 

SS;L1  preferred  over  CO 

4.7 

between  strong  and  weak 
(closer  to  strong) 

SS:L2  preferred  over  CO 

9.9 

absolute 

SS:L1  equal  to  CO:CDTI 

1.6 

equal 

SS:L2  preferred  over  CO:CDTI 

3.3 

weak 

SS:L2  preferred  over  SS;L1 

2.1 

weak 

3.3  Integrated  Results 

The  data  in  the  following  section  discuss  a  subset  of  the  total  data  obtained  from  this  study.  This 
section  contains  the  data  that  were  comparable  between  the  controller  and  pilot  participants. 

3.3.1  Operational  Issues  that  Affect  Shared-Seoaration  Operations 

3. 3. 1.1  Safety  Measmes 

•  Loss  of  Separation  for  Conflicts  Involving  the  NASA  ARC  Simulator. 

In  all  conditions,  the  minimum  separation  distance  was  defined  as  either  five  nm  horizontally 
or  1000/2000  ft  vertically  (as  appropriate).  Pilot  participants  did  not  violate  minimum 
separation  standards  or  receive  any  TCAS  alerts  in  any  flight  segment  of  any  condition. 

•  Loss  of  Separation  for  Conflicts  Involving  WJHTC  Simulation  Pilots. 

During  the  simulation  there  were  two  losses  of  separation  involving  WJHTC  simulation 
pilots.  The  two  separation  violations  occurred  in  SS:L2.  One  violation  involved  a  planned 
conflict  that  resulted  in  a  MSD  of  4.9  nm  while  the  aircraft  were  at  the  same  altitude.  The 
other  was  an  unplanned  conflict  with  a  MSD  of  4.0  nm  while  the  aircraft  were  separated  by 
only  900  ft.  It  must  be  noted  that  controllers  were  not  allowed  to  cancel  free  flight  in  SS:L2. 
Both  separation  losses  were  determined  to  be  due  to  late  maneuvering  (script  execution  error) 
by  WJHTC  simulation  pilots.  In  conclusion  there  was  no  clear  relationship  between  the 
losses  of  separation  and  shared-separation  operations. 
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•  Subjective  Ratings  related  to  Safety. 

Following  each  run,  the  pilot  and  controller  participants  were  asked  to  rate  the  level  of  safety 
using  that  set  of  procedures  and  tools  compared  to  current  flight  operations.  CO  was 
identical  to  current  operations,  so  there  are  no  data  for  this  condition.  See  Figure  30  for 
mean  and  ±1  SEM  bars  for  the  various  conditions.  Inspection  of  the  means  suggest  that  the 
pilots  felt  that  safety  was  somewhat  enhanced  with  the  addition  of  the  new  aircraft 
technologies  and/or  shared-separation  procedures.  The  controllers,  however,  rated  the  level 
of  safety  as  lower  for  the  two  shared-separation  conditions  (SS:L1  and  SS:L2)  when 
compared  to  their  ratings  of  CO:CDTI.  In  addition,  controller  ratings  were  lower  than  pilot 
ratings  in  all  conditions. 


Compromised 
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Figure  30.  Controller  and  pilot  participant  mean  ratings  of  safety. 

3.3. 1.2  Aircraft  Maneuver  Strategies  for  Conflict  Resolution 
•  Aircraft  Maneuver  timing 

For  the  flight  crew  participants,  aircraft  maneuver  time  was  calculated  for  the  two  conditions 
in  which  the  NASA  ARC  simulator  had  the  freedom  to  maneuver  independent  of  ATC 
(SS:L1  and  SS:L2).  For  these  two  conditions,  aircraft  maneuver  timing  was  measured  as  the 
duration  from  the  beginning  of  the  flight  segment  until  the  NASA  ARC  simulator  initiated 
the  first  conflict  avoidance  maneuver.  Table  33  shows  means  and  SDs  for  aircraft 
maneuvering  start  times.  Across  the  shared-separation  conditions,  pilot  participants  started 
maneuvering  to  avoid  conflicting  aircraft  on  average  less  than  5  minutes  into  each  flight 
segment,  which  was  still  roughly  2  to  3  minutes  before  the  CDTI-AL  would  have  indicated  a 
pending  alert.  The  data  suggest  that  the  different  sets  of  pilot  procedures  did  not  impact  the 
timing  of  aircraft  resolution  maneuvers. 
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Table  33.  Aircraft  Maneuvering  Mean  Start  Times  During  Shared-Separation 


SS;L1 

SS:L2 

N  =  6 

M 

SD 

M 

SD 

Aircraft  Maneuver 
Times  (min:sec) 

4:48 

2:05 

4:29 

2:59 

Because  flight  crews  and  controllers  identified  conflicts  quickly  and  initiated  resolution 
maneuvers,  CDTI-AL  alerts  were  not  triggered  for  most  of  the  flight  segments.  Table  34 
presents  the  frequency  of  CDTI-AL  alerts  in  relation  to  resolution  maneuvers.  There  were 
two  instances  when  flight  crews  received  CDTI-AL  alerts  prior  to  their  first  resolution 
maneuver  (once  in  SS:L1,  and  once  in  SS:L2).  In  comparison,  the  flight  crews  received 
CDTI-AL  alerts  after  they  had  initiated  a  resolution  maneuver  once  in  SS:L2,  four  times  in 
SS:L1,  and  once  in  CO:CDTI.  These  alerts  suggest  that  the  maneuvers  the  pilots  initially 
enacted  were  not  sufficient  to  resolve  the  conflict. 


Table  34.  Frequency  of  CDTI-AL  Alerts  in  Relation  to  Maneuver  Start  Times 


Condition 

CDTI-AL  Alerts  Before 
First  Maneuver 

CDTI-AL  Alerts  After 
First  Maneuver 

No  CDTI-AL  Alerts 

CO:CDTI 

0 

1 

8 

SS:L1 

1 

4 

4 

SS:L2 

1 

1 

7 

By  design,  in  SS:L1,  the  NASA  ARC  simulator  could  have  been  imder  direct  air  traffic 
control  (if  the  controller  had  canceled  free  flight)  or  under  shared-separation  procedures. 
Within  SS:L1 ,  there  were  four  flight  segments  in  which  the  NASA  ARC  simulator  received  a 
CDTI-AL  alert  after  the  flight  crew  had  recognized  the  conflict  and  initiated  a  resolution 
maneuver.  In  three  of  these  instances,  the  controllers  did  not  cancel  free  flight,  and  the  flight 
crew  retained  separation  responsibility  imtil  the  conflict  was  resolved  (as  predicted  by  the 
aircraft  alerting  logic).  Interestingly,  in  one  of  those  four  SS:L1  flight  segments,  the 
controller  did  cancel  free  flight  but  not  until  after  the  flight  crew  had  initiated  a  maneuver 
that  -  according  to  the  logic  -  was  sufficient  to  resolve  the  conflict.  In  the  one  case  in  which 
the  CDTI-AL  alert  triggered  before  maneuvering,  the  controller  canceled  free  flight  before 
the  flight  crew  took  any  action. 
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•  Aircraft  Maneuver  Types. 

In  each  flight  segment,  the  pilot  participants  and  the  controllers  could  use  heading,  speed, 
altitude,  or  any  combination  thereof  to  resolve  conflicts.  Table  35  details  the  frequency  of 
maneuver  types  issued  by  controllers  to  either  the  pilots  and/or  the  intruder  simulation  pilot. 
Table  36  lists  the  frequency  of  maneuver  type  initiated  by  the  pilots  in  the  two  shared- 
separation  conditions. 

Table  35.  Frequency  and  Type  of  Maneuvers  Issued  by  Controllers  to  Resolve  Conflicts 
Between  the  Pilot  Participants  and  the  Intruder  Simulation  Pilot 


Condition 

Controller  Issued  Maneuvers  to  Pilot  Participants  and/or 
Intruder  Simulation  Pilot 

Heading 

Altitude 

Speed 

CO 

3 

6 

0 

CO:CDTI 

11 

1 

0 

SS:L1 

4 

1 

1 

SS:L2 

Controllers  could  not  cancel  shared-separation  procedures  in  this  condition 

Table  36.  Frequency  and  Type  of  Maneuvers  Initiated  by  Pilot  Participants  and/or  Intruder 

Simulation  Pilot  to  Resolve  Conflicts 


Condition 

Maneuvers  Initiated  by  the  Pilot  Participants  and/or 
Intruder  Simulation  Pilot  to  Resolve  Conflicts 

Heading 

Altitude 

Speed 

CO 

Flight  crews  could  not  initiate  maneuvers  in  this  condition 

CO;CDTI 

Flight  crews  could  not  initiate  maneuvers  in  this  condition 

SS:L1 

7 

1 

4 

SS:L2 

8 

1 

5 

Interestingly,  in  SS:L1,  controllers  cancelled  free  flight  in  five  of  the  nine  flight  segments. 
They  then  instructed  either  the  pilot  participants  or  the  intruder  simulation  pilot  to  implement 
heading  and  speed  maneuvers  larger  than  those  that  had  been  chosen  by  the  flight  crews. 
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Participants  (controllers  and/or  pilots)  often  used  multiple  sub-maneuvers  to  resolve  a 
conflict.  The  multiple  conflict  resolution  maneuvers  were  typically  enacted  sequentially  to 
resolve  a  single  conflict.  For  example,  a  flight  crew  would  begin  by  using  a  heading  change, 
and  then  add  an  altitude  maneuver  when  it  appeared  that  the  heading  change  would  not 
provide  adequate  aircraft  separation.  Table  37  represents  the  combination  of  maneuvers  used 
by  pilots  and/or  controllers  for  conflict  resolutions. 

Table  37.  Maneuvers  Used  by  Controllers  and/or  Pilot  Participants  to  Resolve  Conflicts 

Involving  the  NASA  ARC  Simulator 


Maneuver 

CO 

CO:CDTI 

SS:L1 

SS:L2 

Speed  only 

- 

- 

- 

- 

Heading  only 

3 

8 

3 

4 

Altitude  only 

6 

1 

- 

- 

Speed  +  heading 

- 

- 

4 

4 

Speed  +  altitude 

- 

- 

1 

Heading  +  altitude 

- 

- 

1 

- 

Speed  +  heading  +  altitude 

- 

- 

1 

- 

For  those  flight  segments  that  did  not  have  an  altitude  change  included  in  the  resolution 
maneuver,  the  horizontal  distances  between  the  NASA  ARC  simulator  and  intruder  aircraft  at 
the  MSD  was  calculated.  Table  38  provides  descriptive  statistics  for  the  MSD  data.  This 
allowed  us  to  investigate  if  there  may  be  differences  between  pilots  and  controllers  in  how 
much  distance  between  aircraft  each  maintained.  Inspection  of  the  means  for  SS;L1  and 
SS:L2  suggest  that  the  flight  crews  tended  to  achieve  less  horizontal  separation  than  when  the 
controller  teams  were  providing  separation  in  CO  and  CO:CDTI.  It  is  difficult  to  interpret 
the  data  for  the  SS:L1  conflicts  when  free  flight  was  canceled  since  these  results  are 
influenced  by  the  fact  that  both  the  pilots  and  controllers  worked  to  resolve  the  conflicts. 
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Table  38.  Descriptive  Statistics  for  Minimum  Horizontal  Distance  for  Conflicts  Involving 
NASA  ARC  Simulator  (Resolved  with  Horizontal  Separation  Only) 


Condition 

Pilot  (only)  Resolved 
Conflicts 

Free  Flight  Cancelled 
Conflicts 

Controller  (only) 
Resolved  Conflicts 

CO 

Flight  crews  could  not  initiate 
maneuvers  in  this  condition. 

Not  Applicable 

Mean  =  10.5 

SD  =  3.5 

N  =  4 

CO:CDTI 

Flight  crews  could  not  initiate 
maneuvers  in  this  condition. 

Not  Applicable 

Mean=  11.0 

5D  =  2.8 

N  =  8 

SS:L1 

Mean  =  8.7 

SD  =  5.7 

N  =  3 

Mean  =  8.2 

SD  =  3.4 

N  =  4 

Controller  involvement  in 
conflict  resolutions  for  this 
condition  was  the  result  of  free 
flight  cancellations. 

SS:L2 

Mean  =  6.2  nm 

SD=  1.1  nm 

N  =  8 

There  were  no  free  flight 
cancellations  by  flight  crews. 
Controllers  could  not  cancel 
free  flight  in  this  condition. 

There  were  no  free  flight 
cancellations  by  flight  crews. 
Controllers  could  not  cancel 
free  flight  in  this  condition 

•  Air<->Ground  Communication 

In  each  of  the  conditions,  an  air^^ground  frequency  was  available  for  all  flight  crews  to 
contact  controllers  or  to  receive  instructions  and  advisories  from  ATC.  All  flight  crews 
conducted  standard  ATC  communications,  such  as  initial  sector  check-ins,  and  any  free  flight 
related  communications  on  the  same  frequency.  Figure  3 1  depicts  the  mean  frequency  of 
airoground  transactions  between  the  pilot  and  controller  participants.  An  air^+ground 
transaction  was  defined  as  all  communication  initiated  by  a  controller  or  pilot  participant. 

For  example,  a  transaction  could  be  all  verbalizations  made  by  the  controllers  and  pilots 
during  the  transfer  of  communication  exchange  or  all  comments  made  about  what  maneuver 
the  flight  crew  was  making  to  resolve  a  CDTI-AL  alert.  It  was  not  necessary  that  a 
transaction  have  an  acknowledgment;  however,  if  there  was  an  acknowledgment,  it  was 
counted  as  part  of  the  same  transaction.  The  frequency  was  a  summation  of  all  the 
transactions  between  pilot  and  controller  participants  during  the  three  flight  segments  for 
each  condition.  The  mean  frequency  for  each  condition  was  then  calculated  as  the  average  of 
the  three  flight  crew  frequencies.  The  data  suggest  that  there  were  more  air<->ground 
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Figure  3 1 .  Mean  frequency  of  air<->ground  transactions. 

transactions  in  SS:L1  compared  to  the  other  three  conditions.  Compared  to  SS:L1,  there 
were  42%  fewer  transactions  in  SS:L2,  54.8%  fewer  transactions  in  CO:CDTI,  and  57% 
fewer  transactions  in  CO.  This  finding  may  be  explained  by  the  fact  that  in  SS:L1,  all  flight 
crews  were  instructed  to  inform  the  controllers  of  all  conflict  avoidance  maneuvers  they  were 
making  prior  to  initiating  them. 

All  flight  crews  were  asked  to  inform  the  controllers  prior  to  the  execution  of  any  and  all  of 
their  aircraft  maneuvers  in  this  condition.  The  procedural  requirement  to  inform  the 
controllers  in  SS:L1  was  also  examined  as  part  of  the  air<->ground  communications.  The 
data  for  SS:L1  indicated  that  within  each  of  these  nine  flight  segments,  the  pilot  participants 
always  notified  the  controllers  of  maneuver  changes  (e.g.,  heading  or  speed  change)  at  least 
once.  For  example,  if  speed  changes  were  made  for  conflict  resolution  more  than  once  in  a 
flight  segment,  the  pilots  always  notified  the  controller  at  least  one  time  that  speed  was  being 
modified.  In  addition,  the  number  of  sub-maneuvers  made  by  the  NASA  ARC  simulator 
when  avoiding  the  intruder  aircraft  was  tallied.  Some  of  these  sub-maneuvers  were 
considered  minor  adjustments,  particularly  in  these  fairly  short  flight  segments,  therefore 
only  those  of  a  relevant  magnitude  were  analyzed  (speed  changes  10  knots  or  greater, 
heading  changes  five  degrees  or  greater,  and  any  altitude  change).  These  data  indicate  that 
the  condition  where  flight  crews  were  required  to  inform  controllers  of  their  intent,  in 
addition  to  informing  at  least  once,  pilot  participants  also  informed  controllers  about  several 
of  their  sub-maneuvers.  They  did  so  for  28  out  of  39  sub-maneuvers.  All  of  these 
notifications  to  the  controllers  occurred  either  just  prior  to  the  execution  of  the  sub-maneuver 
or  just  as  they  began  the  sub-maneuver.  In  SS;L2,  the  pilots  were  instructed  that  informing 
the  controller  of  their  aircraft  maneuvers  was  voluntary.  The  data  revealed  that  for  SS:L2, 
the  pilots  notified  the  controller  of  their  intent  in  only  1  out  of  37  sub-maneuvers  used  to 
avoid  the  intruder  aircraft. 

In  addition  to  the  number  of  air<^ground  communications,  the  average  airo-ground 
transaction  duration  was  calculated.  The  total  transaction  time  was  measured  from  the 
beginning  of  the  first  instruction,  question,  or  comment  made  by  any  of  the  controller  or  pilot 
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participants  to  the  end  of  the  final  communication  on  the  topic.  This  was  meant  to  represent 
the  time  it  would  take  to  handle  a  complete  transaction  because  it  is  common  practice  to  not 
interrupt  an  ongoing  ATC  exchange.  This  time  is  meant  to  represent  the  total  time  required 
to  complete  a  transaction,  therefore  it  included  the  brief  silences  between  pilot  and  controller 
communications.  Figure  32  shows  the  mean  transaction  duration  (and  ±1  SEM  bars)  for  each 
condition.  Mean  air<->ground  transaction  durations  appear  to  be  shorter  in  SS;L2  when 
compared  to  SS:L1  and  COtCDTI  but  not  different  from  CO. 


■cO  BcO:CDTI  BsSiLI  BsS:L2 


Figure  32.  Mean  duration  of  air^ground  transactions. 

The  number  of  missed  communications  was  also  investigated.  This  included  any 
communication  between  the  pilot  and  the  controller  participants,  for  which  there  was  no 
response,  excluding  events  that  are  associated  with  the  flight  crew  being  on  the  wrong  ATC 
frequency.  In  CO,  there  were  39  communication  events.  The  flight  crews  missed  one  of  these 
communications  (2.6%),  and  there  were  no  missed  communications  by  the  controllers.  In 
CO:CDTI,  the  pilot  participants  missed  one  communication  out  of  40  (2.5%)  events.  SS:L1 
had  a  total  of  nine  communications  out  of  74  (5%),  with  five  communications  missed  by  the 
flight  crews  and  four  by  the  controllers.  It  is  important  to  note  that  seven  of  these  came  from 
one  flight  crew/controller  pairing.  All  of  these  missed  communications  were  preceded  by  task- 
related  comments  (i.e.,  instructions,  notifications,  or  queries),  with  the  exception  of  one  call  in 
which  the  controller  asked  the  pilot  participants  if  “they  had  time  for  a  question”  (for  which 
they  received  no  response).  None  of  the  35  communications  in  SS:L2  were  missed. 

3.3.2  Information  Requirements  and  Procedures 

3.3.2. 1  Cancellation  of  Free  Flight  Operations 

In  SS:L1 ,  controllers  were  instructed  they  could  cancel  free  flight  for  one  or  a  pair  of  aircraft  at  a 
time.  During  the  nine  NASA  ARC  simulator  flight  segments  in  this  condition  (three  repetitions 


74 


for  each  of  three  flight  crews),  the  controller  team  canceled  free  flight  in  five  (56%)  of  the  runs. 
In  all  five  runs,  the  NASA  ARC  simulator  and  intruder  aircraft  were  at  the  same  altitude,  and  the 
flight  crews  were  using  heading  and/or  speed  to  attempt  to  resolve  the  conflict.  The  mean 
horizontal  distance  between  conflicting  aircraft  at  the  time  of  free  flight  cancellation  was  13.1 
nm  (SD  =  4.6).  Horizontal  distances  between  conflicting  aircraft  ranged  from  9.0  nm  to  19.9  nm. 

In  the  cases  in  which  free  flight  was  canceled,  the  data  were  examined  to  determine  the 
maneuvering  strategies  for  the  controllers  and  pilot  participants.  In  four  out  of  five  of  the 
cancellations  in  SS:L1,  the  controllers  issued  an  instruction  for  the  same  type  of  maneuver,  but 
of  a  greater  magnitude  than  the  flight  crew  had  already  begun.  For  example,  if  the  flight  crew 
had  started  a  turn  to  relieve  the  conflict,  the  controller  cancelled  free  flight  and  instructed  the 
flight  crew  to  turn  more  sharply.  In  the  fifth  instance,  the  flight  crew  had  attempted  to  use  speed 
to  resolve  the  conflict,  and  the  controller  followed  the  cancellation  with  an  instruction  to  change 
altitude.  In  four  out  of  the  five  flight  segments,  the  controller  returned  the  separation 
responsibility  to  the  aircraft  once  Ae  conflict  had  been  resolved. 

All  pilots  could  cancel  free  flight  operations  at  any  time  in  either  of  the  shared-separation 
conditions  (SS:L1,  SS:L2).  However,  neither  the  simulation  pilots  (per  simulation  design)  nor 
the  pilot  participants  requested  intervention  from  the  controllers. 

3. 3. 2.2  Subjective  Ratines  related  to  Procedures 

Figure  33  shows  controller  and  pilot  participant  ratings  for  the  time  available  to  assure  safe 
aircraft  separation.  Pilot  participants  did  not  conduct  separation  tasks  in  CO  or  CO:CDTI, 
therefore  they  were  not  asked  this  question  for  those  conditions.  In  general,  the  pilots  rated  the 
time  available  for  this  task  as  adequate.  Controller  mean  ratings  for  COrCDTI  appear  to  be 
slightly  higher  than  their  mean  ratings  for  SS:L1  and  SS:L2. 


Too  Much 


Adequite 


1  j 
Too  Little 


CO 

IR-Side 


CO:CDTI 
@  D-Side 


SS:L1 
I  Captains 


SS:L2 
0  First  Officers 


Figure  33.  Controller  and  pilot  participant  mean  ratings  for  time  available  to  assure  safe 
separation. 
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Figure  34  shows  the  controller  and  pilot  participant  ratings  for  the  amount  of  time  available  for 
coordination  and  communication  tasks.  These  tasks  are  likely  to  be  at  least  partially  dependent 
on  when  the  automation  tools  detect  the  conflict.  CDTI-AL  provided  conflict  alerts  about  7 
minutes  prior  to  potential  loss  of  separation  for  the  pilots.  URET  provided  conflict  alerts  about 
13-17  minutes  prior  to  potential  loss  of  separation  for  the  controllers.  Therefore,  one  may  have 
expected  to  see  some  differences  between  the  pilots  and  controllers.  However,  inspection  of  the 
means  reveals  that  there  were  little  differences  between  the  four  conditions  for  the  time  available 
to  complete  required  coordination  and  communication.  The  time  allowed  was  typically  rated  as 
adequate  by  all  participants  with  the  exception  that  the  captains’  rating  for  CO  does  appear  to  be 
higher,  indicating  that  there  was  more  than  adequate  time  for  the  communication  and 
coordination  events. 
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Figure  34.  Controller  and  pilot  participant  mean  ratings  for  time  for  coordination  and 
commimication. 


3.3.3  Controller  and  Pilot  Workload  and  Situation  Awareness 
3.3.3. 1  Controller  and  Pilot  Workload 

Finally,  in  addition  to  the  forms,  workload  ratings  were  gathered  from  both  pilots  and  controllers 
throughout  the  conditions.  Pilot  participants  were  asked  to  rate  their  workload  for  the  flight  at 
the  end  of  each  20-minute  flight  segment  within  each  condition.  Controllers  were  asked  to  rate 
their  workload  every  5  minutes  during  their  1 .5  hour  run  for  each  condition  (previously  referred 
to  as  controller  interval  workload  data).  In  order  to  compare  ratings  over  common  flight 
segments,  controller  interval  workload  ratings  corresponding  to  the  pilot  ratings  were  obtained 
by  taking  the  average  of  the  5-minute  ratings  during  the  time  that  the  NASA  ARC  simulator  was 
on  the  target  sector’s  radio  frequency.  The  mean  ratings  for  the  pilot  workload  responses 
indicated  a  somewhat  higher  workload  in  SS:L1  and  SS:L2  compared  to  pilot  responses  in  CO. 
The  corresponding  average  controller  ratings  seemed  not  to  differ  among  the  various  conditions. 
However,  the  controller  ratings  were  generally  higher  than  the  pilot  ratings  in  CO  and  CO:CDTI, 
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and  similar  in  the  shared-separation  conditions.  Again,  pilot  and  controller  participant  workload 
ratings  were  rather  low  (see  Figure  35). 


Figure  35.  Controller  and  pilot  participant  mean  workload  ratings. 

3.3.3.2  Controller  and  Pilot  Participant  Situation  Awareness 

Figure  36  shows  controller  and  pilot  participant  ratings  for  the  level  of  overall  situation 
awareness  for  the  four  conditions.  In  general,  controllers  rated  their  overall  situation  awareness 
as  high,  and  there  were  only  small  differences  between  the  four  conditions.  Although  there  was 
a  large  amount  of  variance  in  the  flight  crew  data,  inspection  of  the  captain  and  first  officer 
means  indicate  that  the  pilots  may  perceive  the  shared-separation  conditions  as  providing  more 
situation  awareness  when  compared  to  CO  and  COtCDTI. 
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Figure  36.  Controller  and  pilot  participant  mean  ratings  for  overall  situation  awareness. 

4.  DISCUSSION 

Within  each  sub-section,  the  results  are  discussed  in  the  order  corresponding  to  the  three  primary 
objectives  of  the  study. 

4.1  Ground-Side  Discussion 

4.1.1  Operational  Issues  that  Affect  Shared-Separation  Operations 

The  results  indicated  that  controllers  rated  the  safety  for  procedures  compared  to  current 
operations  as  compromised  in  SS:L1  and  SS:L2.  In  their  post-run  ratings  and  comments, 
controllers  expressed  concern  for  safety  while  operating  under  shared-separation  conditions. 
Controllers  frequently  commented  that  pilots  waited  too  long  to  resolve  aircraft  conflicts  and, 
from  the  controller  perspective,  pilot  maneuvers  were  often  barely  adequate  to  ensure  separation. 
Controllers  also  commented  that  pilots  elected  to  fly  their  aircraft  much  closer  to  conflicting 
aircraft  than  controllers  would  normally  allow. 

The  average  duration  of  URET  red  and  yellow  alerts  were  longer  for  SS:L2  compared  to  the 
other  conditions.  These  results  for  the  URET  red  and  yellow  alerts  were  consistent  with 
controller  comments  about  pilots  resolving  conflicts  later  than  controllers.  In  CO  and  CO:CDTI, 
controllers  could  clear  a  URET  alert  by  resolving  the  conflict.  In  SS:L1,  controllers  could 
address  a  URET  alert  by  first  canceling  free  flight  and  then  resolving  the  conflict,  although  they 
allowed  many  pilots  to  continue  on  free  flight  and  solve  their  own  conflicts.  Both  red  and  yellow 
alerts  were  longer  in  SS:L2  most  likely  because  controllers  were  not  allowed  to  cancel  free  flight 
and  also  because  pilot  participants  as  well  as  scripted  simulation  pilot  maneuvers  resolved 
conflicts  later  than  controllers  would  have.  Several  controllers  commented  that  not  being  able  to 
resolve  conflicts  in  a  timely  manner  on  their  ovm  terms  was  stressful. 
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Although  controllers  often  stated  they  felt  safety  was  compromised  in  shared-separation 
operations,  they  rated  the  amount  of  time  available  to  assure  safe  aircraft  separation  and 
complete  required  coordination  as  generally  adequate  in  all  four  conditions.  However,  controller 
ratings  indicated  there  was  slightly  more  time  available  to  assure  safe  aircraft  separation  for 
CO:CDTI  compared  to  SS:L1  and  SS:L2.  In  CO:CDTI  (and  CO),  controllers  tended  to  resolve 
aircraft  conflicts  early,  whereas  conflicts  tended  to  be  resolved  later  in  SS:L1  and  SS:L2.  When 
aircraft  conflicts  were  resolved  relatively  later,  controllers  may  have  felt  that  there  was  slightly 
less  time  available  to  assure  safe  aircraft  separation. 

The  MSDs  for  altitude-resolved  planned  conflicts  involving  simulated  aircraft  flown  by  WJHTC 
simulation  pilots  were  much  greater  for  CO  and  CO:CDTI  compared  to  SS:L1  and  SS:L2.  The 
MSDs  for  vector-resolved  conflicts,  however,  were  not  very  different  between  the  four 
conditions.  The  different  results  for  altitude-resolved  and  vector-resolved  conflicts  were  likely 
due  to  the  different  methods  for  computing  the  MSDs.  It  is  important  to  understand  that  the 
MSD  for  an  altitude-resolved  conflict  did  not  represent  the  absolute  closest  horizontal  distance 
between  the  conflicting  aircraft.  The  MSD  represents  only  the  horizontal  distance  when  the 
aircraft  pair  have  not  achieved  altitude  separation.  Most  altitude-resolved  conflicts  passed  very 
close  to  each  other  horizontally,  but  altitude  ensured  safe  separation.  In  contrast,  the  MSD  for  a 
vector-resolved  conflict  represents  the  absolute  closest  horizontal  distance  between  the 
conflicting  aircraft.  In  the  case  of  altitude-resolved  conflicts,  early  actions  produced  greater 
MSDs  relative  to  later  actions.  Therefore,  the  MSD  results  for  altitude-resolved  conflicts  were 
consistent  with  the  controller  strategy  to  resolve  conflicts  early  and  allow  a  greater  separation 
distance  when  controlling  aircraft  on  their  own  terms. 

Another  indication  that  controllers  felt  that  safety  was  compromised  during  shared-separation 
operations  was  they  cancelled  free  flight  for  12  of  the  39  SS:L1  planned  conflicts  (31%) 
involving  WJHTC  simulation  pilots  (all  aircraft  other  than  the  NASA  ARC  simulator  and 
intruder  aircraft).  In  some  cases,  controllers  cancelled  free  flight  after  one  of  the  pilots  initiated  a 
conflict-resolving  maneuver  suggesting  that  controllers  were  not  always  certain  that  pilot 
maneuvers  would  be  effective. 

4.1.2  Controller  Information  Requirements  and  Procedures 

Controllers  rated  the  amount  of  information  necessary  to  resolve  conflicts  and  the  look-ahead 
time  of  the  URET  conflict  alerts  as  adequate  in  all  four  conditions.  However,  some  controllers 
commented  that  URET  alerts  were  too  early  in  some  cases  and  too  late  in  others.  In  general, 
controllers  indicated  that  URET  was  beneficial  and  provided  enough  information  to  help  them 
identify  and  resolve  aircraft  conflicts  in  both  current  operational  conditions  and  shared- 
separation  conditions.  Although  controllers  felt  that  they  had  enough  information  to  resolve 
conflicts,  many  controllers  commented  that  they  needed  pilot  intent  information  sooner  during 
shared-separation  operations. 

Controllers  rated  the  procedures  for  SS:L1  and  SS:L2  as  generally  not  helpful  for  performing 
their  jobs.  Controller  comments  focused  on  perceived  reductions  in  safety.  Controllers  reported 
that  they  did  not  feel  completely  “in  control”  of  the  traffic  situation,  and  this  was  very  stressful. 
Controllers  stated  they  felt  that  shared-separation  procedures  put  them  in  a  “reactive”  control 
mode  instead  of  allowing  them  to  be  “proactive”  and  use  their  typical  planning  skills.  Other 
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investigators  had  similar  findings  (Corker  et  al.,  2001).  Controllers  were  concerned  that  pilots 
would  cancel  free  flight  in  a  conflict  situation  that  would  be  impossible  for  controllers  to  resolve 
in  time.  Controllers  stated  they  felt  that  the  pilot  style  for  resolving  conflicts  compromised 
safety.  As  previously  stated,  controllers  felt  that  pilots  waited  too  long  to  resolve  conflicts  and 
maneuvered  too  closely  to  conflicting  aircraft. 

Controllers  formed  more  URET  trial  plans  in  CO  and  CO:CDTI  compared  to  SS:L1  and  SS:L2. 
Controllers  use  the  URET  trial-planning  feature  as  a  tactical  tool  to  help  determine  if  a  proposed 
control  action  would  impact  other  aircraft.  In  CO  and  CO:CDTI,  controllers  were  completely 
responsible  for  aircraft  separation,  which  was  likely  why  controllers  frequently  used  trial 
planning.  In  SS:LI,  controllers  may  have  used  trial  planning  less  because  procedures  allowed 
pilots  maneuvering  flexibility  and  thereby  reduced  time  for  controllers  to  plan  and  resolve 
conflicts  (including  trial  planning)  iftwhen  they  canceled  free  flight.  In  addition,  during  SS:L1 
and  SS:L2,  controllers  reported  that  they  did  not  feel  completely  “in  control”  of  the  traffic 
situation.  In  SS:L2  especially,  controllers  may  have  felt  that  trial  planning  was  not  very  useful 
because  they  were  not  able  to  cancel  free  flight  and  resolve  aircraft  conflicts.  These  changes  in 
their  roles  and  responsibilities  may  also  explain  why  controllers  used  trial  planning  less  during 
shared-separation  operations. 

In  the  initial  study  briefing,  the  researchers  explained  that  controllers  may  monitor  air^air 
conununications  in  SS:L1  and  SS:L2  as  little  or  as  much  as  they  wanted.  The  researchers 
suggested  that  controllers  should  try  monitoring  during  the  training  runs  to  determine  if  it  was 
useful  during  shared-separation  operations.  Each  R-side/D-side  controller  team  decided  that  the 
D-side  controller  should  monitor  the  air<->air  frequency.  D-side  controllers  reported  that  they 
always  monitored  air<->air  communications.  For  half  the  teams,  the  R-side  controller  also 
decided  to  monitor  air<->air  communications  and  reported  that  they  always  monitored  as  well. 
The  R-side  and  D-side  controllers  who  monitored  the  air^air  frequency  reported  that  it  was 
useful.  The  R-side  controllers  who  decided  not  to  monitor  the  air<->air  frequency  commented 
that  it  was  distracting  to  listen  to  both  air<-»air  and  air<-»ground  commxmications  at  the  same 
time,  particularly  during  SS:L1 .  These  R-side  controllers  stated  that  it  was  sufficient  that  their 
D-side  team  member  monitor  air<->^air  communications  and  report  to  them  any  necessary 
information.  Additionally,  controllers  suggested  that  adding  a  speaker  for  air^air 
communications  may  have  been  helpful. 

4.1.3  Controller  Workload  and  Situation  Awareness 

Several  different  measures  and  techniques  in  the  study  assessed  controller  workload.  The 
researchers  employed  a  form  to  collect  post-run  workload  ratings  and  the  WAK  tool  to  collect 
interval  workload  ratings  as  controllers  worked  the  traffic.  In  addition,  experienced  air  traffic 
control  specialists  participated  as  EOs  in  the  simulation  and  provided  ratings  of  controller 
physical  taskload,  a  measure  related  to  workload.  Finally,  controller  workload  was  a  major  topic 
of  discussion  in  the  post-run  and  exit  debriefings. 

The  results  indicated  that  controller  post-run  ratings  of  mental  and  overall  workload  were  higher 
in  SS:Ll  compared  to  the  other  conditions.  Controllers  commented  that  having  to  monitor 
aircraft  conflicts  that  they  would  have  resolved  earlier  added  a  great  deal  to  their  mental 
workload  and  increased  their  stress  level.  Controllers  reported  that  under  current  operating 
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procedures,  they  routinely  resolve  conflicts  early  and  then  they  do  not  have  to  think  about  the 
conflicts  any  further.  However,  shared-separation  operations  seemed  to  change  their  monitoring 
style  and  forced  them  to  keep  more  situations  in  mind  than  they  do  normally.  Additionally, 
controllers  repeatedly  stated  that  having  to  develop  multiple  alternative  plans  depending  upon 
what  pilots  might  do  increased  their  workload.  Controller  ratings  indicated  that  most  of  the 
additional  workload  of  shared-separation  operations  was  mental,  and  there  was  not  much  more 
physical  workload. 

Although  controllers  reported  that  mental  and  overall  workload  were  higher  in  SS:L1,  their 
ratings  indicated  that  their  absolute  workload  levels  were  not  much  above  moderate.  The  traffic 
scenarios  developed  for  the  simulation  were  not  very  complex  compared  to  actual  traffic  in  these 
two  sectors.  By  design,  the  simulation  depicted  rather  sterile  traffic  situations  with  very  few 
transitioning  aircraft  climbing  or  descending  into  adjacent  airspace.  Additionally,  mixed 
equipage  was  not  addressed;  there  were  no  restrictions  due  to  active  SUA  or  in-trail 
requirements,  and  no  adverse  weather  conditions  were  in  effect.  Overall,  the  traffic  scenarios 
were  moderately  busy,  had  lower  complexity,  and  did  not  cause  high  workload  for  controllers. 

Controller  post-run  workload  ratings  were  generally  the  same  or  lower  in  SS:L2  compared  to 
SS:L1 .  In  one  regard,  controller  workload  might  be  expected  to  increase  in  SS:L2  because 
controllers  did  not  receive  pilot  intentions  of  their  maneuvers.  However,  there  were  critical 
differences  between  SS:L1  and  SS:L2  that  may  have  reduced  controller  workload.  In  contrast  to 
SS:L1,  there  were  no  free  flight  cancellations  in  SS:L2  because  controllers  were  not  allowed  to 
cancel  free  flight  and  no  pilots  requested  cancellation.  Controllers  were  not  permitted  to  deliver 
any  control  instructions  in  SS:L2,  therefore  there  were  fewer  conununications  to  pilots  and 
controllers  reported  that  they  only  sometimes  evaluated  alternative  plans  to  assure  aircraft 
separation.  These  artifacts  may  have  resulted  in  lower  controller  workload  for  SS:L2. 

The  interval  workload  ratings  collected  using  the  WAK  tool  were  slightly  different  from  the 
controller  post-run  workload  ratings.  In  general,  the  interval  workload  ratings  did  not  indicate 
any  significant  differences  between  the  four  conditions.  The  differences  in  workload  for  the 
interval  and  post-run  workload  rating  techniques  may  be  because  the  techniques  rely  on  slightly 
different  workload  information  (instantaneous  versus  overall).  EO  ratings  of  controller  physical 
taskload  were  consistent  with  the  results  of  the  controller  post-run  workload  ratings  and 
controller  comments  about  higher  workload  in  SS:L1 . 

Controllers  made  fewer  ground— >air  and  land  line  PTTs  in  SS:L2  compared  to  the  other 
conditions.  A  content  analysis  of  these  communications  may  have  been  able  to  specify  the 
reason  for  these  differences  but  this  could  not  be  accomplished  under  the  time  constraints  of  this 
study.  These  results  are  likely  due  to  the  procedural  differences  in  SS;L2.  In  both  CO  and 
CO:CDTI,  controllers  made  ground->air  and  land  line  PTTs  to  resolve  all  aircraft  conflicts.  In 
SS;L1,  controllers  cancelled  free  flight  for  many  aircraft  and  made  ground->air  and  land  line 
PTTs  to  resolve  the  conflicts.  In  SS:L2,  however,  controllers  were  not  allowed  to  cancel  free 
flight  and  no  related  instructions  (via  ground— >air  or  land  line  PTTs)  were  made  to  actively 
maneuver  aircraft.  In  addition,  SS:L2  had  no  scripted  pilot  inquiries  or  intent  information  relays 
(and,  therefore,  no  controller  responses  or  acknowledgments).  These  factors  likely  contributed 
to  the  decrease  in  SS:L2  PTTs  compared  to  the  other  conditions. 
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Finally,  controller  ratings  of  situation  awareness  were  high  and  not  significantly  different  in  the 
four  conditions.  Controller  comments  about  changing  their  monitoring  style,  keeping  more 
situations  in  mind,  and  developing  alternative  plans  during  shared-separation  operations  seem  to 
suggest  that  controller  situation  awareness  should  have  declined.  However,  during  the 
debriefing,  controllers  stated  that  despite  increasing  mental  workload,  they  had  sufficient 
information  and  were  able  to  maintain  their  awareness  of  the  traffic  situations.  This  information 
suggests  that  it  was  not  because  of  reduced  controller  situation  awareness  that  controllers  felt 
safety  was  compromised  in  the  shared-separation  conditions.  The  reasons  for  controller 
concerns  about  safety  during  shared-separation  operations  seem  directly  related  to  pilot  conflict 
resolution  strategies  and  maneuvers. 

In  summary,  controllers  felt  that  safety  was  compromised  in  the  shared-separation  conditions 
simulated  in  the  present  study.  The  results  from  the  ground-side  indicated  that  the  factors 
controllers  perceived  to  reduce  safety  were  unpredictable  and  barely  adequate  pilot  conflict 
resolution  maneuvers  that  raised  controller  workload,  increased  controller  stress,  and  reduced  the 
time  available  for  controllers  to  intervene.  In  general,  controllers  felt  that  pilots  waited  too  long 
to  resolve  conflicts  and  maneuvered  too  closely  to  conflicting  aircraft  during  the  shared- 
separation  conditions.  Controllers  discussed  their  concerns  about  shared-separation  operations 
with  the  research  team  and  offered  valuable  insight  into  their  feelings  and  reactions  to  the 
simulation.  This  feedback  is  important  to  understand  the  issues  that  need  to  be  addressed  should 
the  FAA  consider  implementing  a  shared-separation  concept  in  the  future. 

4.2  Air-Side  Discussion 

The  discussion  of  the  air-side  data  refer  only  to  pilot  participant  information.  Data  concerning 
simulation  pilots  (WJHTC  and  NASA)  are  not  included. 

4.2.1  Operational  Issues  that  Affect  Shared-Seoaration  Operations 

4.2. 1.1  Safety  measures 

To  determine  if  the  conditions  had  an  impact  on  pilot  participant  behavior  regarding  flight  safety, 
the  following  measures  were  analyzed:  loss  of  separation  and  flight  crew  ratings  of  safety.  There 
were  no  losses  of  separation  between  the  NASA  ARC  simulator  and  other  aircraft,  defined  as 
less  than  5  nm  horizontal  or  1000/2000  ft  vertical  separation  (as  appropriate),  during  any  of  the 
flight  segments.  Therefore,  separation  was  maintained  regardless  of  who  (flight  crew  or 
controller)  retained  separation  authority.  Keep  in  mind  that  there  were  only  three  flight  crews 
who  flew  36  flight  segments.  In  previous  studies  using  a  similar  CDTI-AL,  a  small  percentage 
of  runs  did  result  in  a  loss  of  separation  when  pilots  attempted  to  maintain  separation  without 
direct  involvement  of  the  controller.  This  included  work  in  both  a  full-motion  simulator  (Lozito 
et  al.,  2000)  and  in  part-task  simulation  (Cashion  &  Lozito,  2000). 

Pilots  also  reported  that  for  flight  safety,  SS:L2  was  absolutely  preferred  over  CO  and  strongly 
preferred  over  COiCDTI.  The  pilots  seemed  to  feel  safety  was  improved  when  they  had  more  of 
a  role  and/or  more  control  over  conflict  management.  Also  not  surprising  was  that  pilots 
reported  a  preference  for  the  CDTI-AL  condition  over  the  condition  without  the  CDTI-AL. 
Perhaps  they  felt  safety  was  enhanced  when  they  could  see  the  surrounding  traffic  even  when  the 
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controller  retained  authority.  In  addition,  the  CDTI-AL  might  have  provided  them  with  context 
as  to  why  they  received  a  course/altitude  deviation  or  enabled  them  to  verify  the  controller’s 
avoidance  solution.  Their  preference  may  also  reflect  improved  traffic  awareness. 

4.2. 1.2  Aircraft  Shared-Senaration  Performance 

Another  safety  measure  was  the  time  available  for  the  pilot  participants  to  detect  conflicting 
aircraft.  The  flight  crews  detected  conflicts  early  on  in  the  flight  segments,  well  before  the  onset 
of  CDTI-AL  alerts.  This  is  similar  to  findings  from  other  studies  of  pilots  in  a  shared-separation 
environment  (Lozito  et  al.,  2000).  Flight  crews’  self-reported  data  corroborates  the  timing  data 
as  they  rated  the  conflicts  easy  to  identify  across  all  of  the  CDTI-AL  conditions.  The  pilots  also 
rated  the  CDTI-AL  as  being  effective  for  shared-separation.  It  is  important  to  note  that  flight 
crews  were  flying  en  route  flight  segments,  generally  considered  a  low  workload  environment 
under  normal  conditions.  Pilot  participants  also  reported  spending  a  considerable  amount  of  time 
visually  scanning  the  CDTI-AL.  It  should  be  noted  that  the  CDTI-AL  features  in  this  study 
existed  on  the  Navigation  Displays  for  the  captain  and  first  officer,  and  that  these  displays 
contain  a  great  deal  of  navigation  data  as  well  as  traffic  information. 

4.2. 1.3  Flight  Efficiency  Measures 

Although  fuel  bum  measurements  could  not  be  assessed  in  this  study  due  to  the  short  flight 
segments,  the  pilots  compared  each  of  the  conditions  to  each  other  to  determine  which  conditions 
were  preferred  for  flight  efficiency.  When  asked  which  conditions  they  felt  were  most  efficient 
for  their  flight  segments,  the  pilots  generally  rated  the  shared-separation  conditions  as  more 
efficient  than  CO  and  CO:CDTI.  Not  surprising,  these  ratings  were  strongest  when  comparing 
SS:L2,  the  shared-separation  condition  in  which  the  controllers  could  not  intervene,  to  CO  and 
COiCDTI.  The  pilots  absolutely  preferred  the  condition  that  provided  them  with  the  most 
maneuver  flexibility. 

4.2. 1 .4  Flight  Crew  Communication 

Pilot  participants  generally  reported  that  the  air<->air  communications  were  useful  to  monitor. 
Pilots  also  reported  they  had  an  adequate  amount  of  time  available  for  communication. 

However,  they  only  contacted  the  intmder  aircraft  in  7  out  of  18  conflict  situations  during  the 
shared-separation  conditions.  They  also  rarely  contacted  other  aircraft  for  any  reason.  These 
results  are  somewhat  surprising,  as  previous  research  has  found  that  flight  crews  in  a  shared- 
separation  environment  often  contacted  the  intruder  aircraft  to  confirm  their  intentions  or  query 
them  regarding  maneuvering  (Lozito  et  al.,  2000).  In  those  previous  studies,  controllers  were  not 
actively  controlling  aircraft  in  any  of  the  flight  segments.  Perhaps  the  presence  of  a  controller  as 
a  backup  to  provide  resolutions  and  the  frequency  congestion  generated  by  communications  fi’om 
the  controller  and  other  simulation  pilots  discouraged  the  use  of  the  air<->air  frequency. 

4.2.2  Flight  Crew  Information  Requirements  and  Procedures 

4.2.2. 1  Flight  Crew  Information  Requirements 

Form  data  revealed  that  the  pilot  participants  found  the  CDTI-AL  and  the  associated  conflict 
alert  logic  quite  effective  for  safe  operations.  They  reported  that  there  was  only  slight  confusion 


83 


regarding  who  had  separation  authority  during  the  flight  segments.  However,  it  should  be  noted 
that  a  few  comments  by  the  pilots  during  data  collection  suggested  that  there  may  have  been 
some  confusion  regarding  procedures  and  separation  authority.  As  an  example,  one  pilot  asked 
the  other  if  they  were  still  in  free  flight  during  one  of  the  flight  segments.  Participants  also 
indicated  that  the  amount  of  information  on  the  CDTI-AL  was  adequate  to  identify  and  resolve 
conflicts  and  that  the  timing  of  the  conflict  alert  was  adequate  for  strategic  separation.  The  flight 
crew  spent  a  considerable  amount  of  time  monitoring  the  CDTI-AL  with  the  percentage  of  time 
increasing  as  the  pilot  responsibility  for  separation  increased.  This  percentage  would  probably 
increase  with  the  presence  of  weather,  which  was  omitted  in  this  study.  The  amount  of 
heads-down  time  may  be  a  subject  of  concern,  particularly  if  there  were  abnormal  events  during 
the  flight.  The  reduced  amount  of  time  pilots  spent  looking  out  the  window  or  monitoring  other 
displays  will  need  to  be  considered  in  the  development  of  new  display  technology. 

The  temporal  predictors  on  the  CDTI-AL  seemed  to  aid  the  pilots  considerably  because  they 
were  selected  on  and  in  use  most  of  the  time.  The  map  range  on  the  CDTI-AL  was  set  to  160  nm 
a  majority  of  the  time,  followed  by  the  80  nm  range.  These  data  are  consistent  with  ADS-B  range 
capability,  and  also  with  prior  research  regarding  CDTI-AL  map  range  settings  in  free  flight 
(Lozito,  McGann,  Mackintosh,  &  Cashion,  1997;  Lozito  et  al.,  2000).  Map  range  settings  would 
also  probably  be  affected  by  weather  because  pilots  would  be  looking  for  weather  cells  at 
extended  ranges  throughout  the  flight. 

4.2.3  Flight  Crew  Workload  and  Situation  Awareness 

4.2.3. 1  Flight  Crew  Workload 

Two  methods  were  employed  to  assess  flight  crew  workload,  namely  Subjective  Workload 
Ratings  and  AHP  Workload  Ratings.  Both  measures  required  the  participants  to  self-assign 
workload  ratings.  The  Subjective  Workload  Ratings  were  collected  after  each  flight  segment  and 
evaluated  only  in  that  condition.  The  AHP  Workload  Ratings  were  gathered  after  completing  all 
runs  and  provided  a  comparative  assessment  of  conditions.  Workload,  as  reported  by  the  pilots, 
averaged  from  very  low  to  low. 

The  highest  relative  workload  was  reported  in  one  of  the  shared-separation  conditions  (SS:L1). 

In  the  conditions  in  which  the  flight  crews  flew  using  current  operational  procedures  (CO  and 
COiCDTI),  the  tasks  were  fairly  simple  for  the  crews.  The  shared-separation  environment  in 
which  the  crews  were  required  to  inform  the  controller  of  maneuvers  required  more 
communication.  The  workload  ratings  did  reveal,  however,  that  even  with  these  new  tasks,  the 
flight  crew  workload  was  still  perceived  as  low.  Furthermore,  the  flight  crews’  AHP  ratings 
indicated  a  strong  preference  for  SS:L2  over  CO  for  reducing  their  workload.  The  rest  of  the 
conditions  were  rated  equal  in  value  for  reducing  workload.  Perhaps  the  pilots  felt  that  when 
they  had  more  control  over  their  own  operations,  their  workload  was  reduced.  This  finding  is 
interesting  when  their  physical  and  mental  workloads  should  have  increased  with  the  activities 
associated  with  the  hipest  level  of  shared-separation  operations.  Another  explanation  is  they 
may  just  prefer  to  be  more  involved  in  maintaining  separation  and  therefore  always  rated  SS:L2 
as  the  most  favorable  condition. 
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4.2.32  Flight  Crew  Situation  Awareness 


The  AHP  situation  awareness  data  were  slightly  more  conclusive.  For  situation  awareness,  SS:L2 
was  absolutely  preferred  over  CO  (similarly,  workload  indicated  a  strong  preference).  The 
results  also  indicated  a  nearly  strong  preference  for  SS;L1  over  CO  for  maintaining  situation 
awareness.  Interestingly  CO:CDTI  was  recorded  as  having  only  a  weak  level  of  dominance  for 
preference  over  CO. 

The  pilot  participants  apparently  believed  that  there  was  a  relationship  between  the  operations  of 
shared-separation  and  situation  awareness.  Although  they  were  provided  with  the  CDTI-AL  in 
CO:CDTI,  the  flight  crews  did  not  think  that  this  condition  offered  much  more  situation 
awareness  than  not  having  the  information  at  all.  However,  when  allowed  the  opportunity  to  use 
the  CDTI-AL  to  separate  themselves,  as  in  SS:L1  and  SS:L2,  they  seemed  to  feel  that  their 
situation  awareness  was  enhanced.  The  opportunity  to  use  the  CDTI-AL  information  in  the 
shared-separation  conditions,  along  with  the  responsibility  that  those  procedures  entail,  seemed 
to  increase  their  perception  of  their  situation  awareness. 

In  general,  the  pilot  participants  had  favorable  comments  regarding  the  air-side  tool  and 
procedures  associated  with  the  shared-separation  conditions.  There  were  no  observable 
compromises  in  safety.  Pilots  did  report  higher  workload  for  the  shared-separation  conditions 
compared  to  the  CO  and  CO:CDTI  conditions,  but  the  workload  ratings  never  reached  above  the 
moderate  level.  However,  the  ability  to  generalize  from  these  data  is  limited  by  the  small  sample 
size,  as  well  as  the  relatively  ideal  operational  context  of  the  simulation  itself  (i.e.,  no  weather  or 
abnormal  events). 

4.3  Integrated  Discussion 

This  section  discusses  findings  about  the  shared-separation  concept  from  an  integrated 
perspective.  Table  39  summarizes  interesting  results  of  the  integrated  data  for  easy  referencing. 

4.3.1  Operational  Issues  that  Affect  Shared-Seoaration  Operations 

4.3. 1.1  Safety  Measures 

The  pilot  and  the  controller  participants  appeared  to  have  different  views  regarding  the  level  of 
safety  for  shared-separation  operations.  Pilots  seemed  to  feel  that  safety  was  not  compromised 
using  the  new  tools  and  procedures,  but  the  controllers  appeared  to  have  more  concerns  with  the 
operational  concept.  However,  it  is  interesting  that  the  pilots  essentially  rated  COrCDTI  as 
having  the  same  level  of  safety  as  CO.  They  did  not  appear  to  make  a  distinction  in  safety  ratings 
between  the  current  operations  with  or  without  a  CDTI. 

In  SS;L2,  pilot  tasks  and  responsibilities  increased  significantly  (e.g.,  monitoring  CDTI-AL, 
airoair  communications,  detecting  and  resolving  conflicts)  beyond  their  normal  activities 
therefore,  interestingly,  they  still  perceived  SS:L2  as  a  safer  operation.  The  plausible 
explanation  is  that  shared-separation  procedures  that  required  pilots  to  look  ahead  for  conflicts 
and  separate  themselves  may  have  increased  their  situation  awareness  and  “pro-active” 
involvement  in  their  flight  planning.  However,  controllers  felt  that  CO  and  COiCDTI  were  the 
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Table  39.  Summary  of  Integrated  Results 


Controller  Participants:  Ground-Side 

Pilot  Participants:  Air-Side 

CO 

COrCDTI 

SS:L1 

SS:L2 

CO 

COrCDTI 

SSrLl 

SS:L2 

Overall  Workload 

Higher 

AHP  and  post  run  rating  results  are  not  consistent'^ 

Mental  Workload 

Higher 

Higher 

Physical  Workload 

Higher 

Coordination 

workload 

No  difference 

Highest 

Communication 

workload 

No  difference 

No  difference 

Situation 

Awareness 

No  difference 

Higher 

0 

0 

0 

2 

0 

0 

0 

0 

Cancellation  of 
participant  pilot 
free  flight 

N/A 

N/A 

5  out  of  9 
times  by 
controller 

N/A 

N/A 

None  by 
participant 
pilot 

None  by 
participant 
pilot 

Average  horizontal 
separation  distance 
( participant  pilot 
conflicts  only) 

10.5  nm 

1 1.0  nm 

8.2  nm 

N/A 

N/A 

N/A 

8.7  nm 

6.2  nm 

Air  4->air 
frequency 
monitoring 

N/A 

_ 

N/A 

100%  of  D-sides 
monitored,  but  only 
50%  of  R-sides 
monitored. 

N/A 

N/A 

Did  not  always 
monitor.  Monitoring 
was  moderately  useful 

Level  of  Safety 
ratings 

Unchanged 

Compromised 

Strong 

preference 

Time  available  to 
complete  all 
communication 
and  coordination 

All  Adequate 

Captains 

indicated 

more 

than 

adequate 

N/A 

Adequate 

Adequate 

Time  available  to 
assure  safe 
separation 

Adequate 

Slightly  below 
adequate 

N/A 

N/A 

Adequate 

Adequate 

AHP  results  indicate  reduced  workload  under  SSL2  as  compared  with  CO  where  as  subjective  ratings  indicate 
higher  workload  under  SSL2  and  SSLl  as  compared  with  CO  and  CO:CDTI. 
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safer  operations  and  reported  that  SS:L1  and  SS:L2  compromised  safety.  Like  the  pilots,  the 
controllers  did  not  make  a  distinction  in  their  safety  ratings  between  the  current  operations  with 
the  presence  or  absence  of  a  CDTI.  However,  there  do  appear  to  be  some  concerns  with  the 
presence  of  the  new  procedures  when  coupled  with  the  CDTI  technology.  The  fact  that 
controllers  cancelled  free  flight  in  SS:L1  and  indicated  that  they  would  have  cancelled  in  SS:L2 
reinforces  their  concern  for  safety.  Thus,  the  controllers  apparently  felt  that  the  safety 
compromise  may  reside  in  the  new  procedures  and  not  in  the  new  technologies  and  information 
provided  to  the  flight  crews. 

Conflict  Resolution  Strategies 

There  appear  to  be  some  interesting  differences  between  the  pilot  and  the  controller  participants 
with  respect  to  maneuvering  strategies.  Both  controllers  and  pilots  often  used  heading  as  part  of 
their  resolution  strategy  when  responsible  for  separation.  When  responsible  for  their  own  conflict 
resolution  (in  SS:L1  and  SS:L2),  pilots  may  have  found  heading  changes  advantageous  because 
their  CDTI-AL  and  navigation  display  did  not  provide  as  much  information  in  the  vertical 
direction  (±  4100  ft  only).  This  could  explain  why  controllers  were  more  likely  to  use  altitude 
changes  in  CO  because  pilots  did  not  have  CDTI-AL.  Perhaps  controllers  were  likely  to  work 
with  heading  changes  in  CO: CDTI  and  SS:L1  because  they  were  aware  that  pilots  had  CDTI-AL 
that  gave  them  a  traffic  view  and  conflict  prediction  extended  beyond  the  TCAS  range.  Another 
notable  observation  was  that  the  controllers  rarely  used  speed,  whereas  pilots  frequently  used 
speed  changes.  Because  pilot  participants  were  only  responsible  for  their  own  flight,  they  may 
have  had  more  time  for  conflict  management  (versus  controllers  being  responsible  for  all  aircraft 
in  the  entire  sector).  For  this  reason,  they  may  have  attempted  maneuvering  resolutions,  which 
take  more  time  to  enact  and  monitor  (e.g.,  speed  changes)  or  even  more  complicated  strategies 
(e.g.,  speed  and  heading  changes  together).  This  may  also  explain  why  pilots  seemed  to  use 
more  multiple  sub-maneuvers  for  conflict  resolution  than  controllers  do.  Corker  et  al.  (2001) 
also  found  more  prevalent  use  of  a  single  maneuver  resolution  by  controllers  for  shared- 
separation  conflict  management. 

Interestingly,  controllers  tended  to  resolve  conflicts  earlier  than  pilots  do.  It  may  be  that  the 
controllers  were  resolving  conflicts  quickly  so  they  could  continue  to  monitor  other  aircraft. 

Pilots  resolved  conflicts  rather  late  (from  the  controllers  perspective)  thereby  creating  minimum 
impact  on  their  flight  plan  and  efficiency.  It  must  be  noted  that  no  pilot  participants  violated 
separation  minima,  and  they  tended  to  maneuver  before  the  airborne  alert  logic  triggered  display 
changes  to  notify  the  crew  of  the  conflict.  This  is  consistent  with  previous  findings  from  similar 
research  in  the  area  of  free  flight  (Cashion  et  al.,  1997).  Another  explanation  of  these  differences 
may  possibly  be  due  to  controller  roles  and  responsibilities  for  the  entire  sector  versus  pilot 
responsibility  for  only  their  individual  aircraft.  As  stated  previously,  pilots  may  have  also  had  a 
bit  more  time  to  manage  their  own  traffic  conflict,  particularly  in  an  en  route  flight  segment  in 
which  there  are  no  abnormal  circumstances.  The  apparent  preference  for  controllers  to  solve 
conflicts  early  was  also  reflected  in  their  lower  safety  ratings  during  SS:L1  and  SS:L2. 
Additionally,  controllers  tended  to  cancel  free  flight  if  the  pilots  didn’t  initiate  a  maneuver  and/or 
didn’t  inform  controllers  about  their  maneuver  prior  to  a  point  where  the  controller  would  have 
cancelled  free  flight.  Furthermore,  when  controllers  cancelled  free  flight,  they  tended  to  only 
change  the  magnitude  of  the  resolution  (e.g.,  increased  rate  of  turn)  that  was  being  executed  by 
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pilots.  This  may  be  because  the  participant  pilots’  strategies  were  perceived  as  correct  by  the 
controllers,  but  the  magnitude  was  simply  not  enough. 

The  data  indicating  MSDs  for  conflicts  involving  the  pilot  participants  were  also  explored. 

These  data  revealed  that  controllers  appeared  to  prefer  greater  separation  distances  (than  pilots) 
while  actively  controlling  traffic.  In  the  conditions  in  which  the  controllers  had  full  separation 
authority,  CO  and  CO:CDTI,  the  MSDs  on  average  were  10.5  nm  and  1 1  nm  respectively.  The 
instances  when  the  pilots  had  separation  responsibility  for  themselves,  in  SS:L1  (unless  free 
flight  was  cancelled)  and  always  in  SS:L2,  the  average  MSDs  were  8.7  nm  and  6.2  nm. 
Controllers  seem  to  be  more  comfortable  with  a  larger  buffer,  perhaps  again  indicating  the  need 
to  manage  their  tasks  to  allow  time  for  the  other  jobs  they  must  perform.  It  should  be  recognized 
that  there  is  much  variance  in  these  data,  so  it  appears  that  across  the  relatively  small  sample  size 
for  the  pilot  and  controller  participants,  there  are  some  differences  that  may  be  attributable  to 
individual  styles  or  strategies  for  controlling  and  flying  aircraft.  The  observed  pilot  participant 
strategies  may  have  also  been  influenced  by  their  primary  domain  experience  (all  oceanic  pilots 
were  used)  and  the  fact  that  pilots  are  not  trained  as  are  controllers,  to  manage  and  resolve 
conflicts. 

4.3 . 1 .2  AiroGround  Communication 


The  air<->ground  communications  data  may  reveal  more  distinctions  between  the  conditions. 

The  shared-separation  condition  in  which  the  controllers  were  able  to  cancel  free  flight  (SS:L1) 
seemed  to  have  a  pattern  indicating  a  general  increase  in  communications  between  the 
controllers  and  the  flight  crew  participants.  As  noted,  this  is  partly  an  artifact  of  the  requirement 
for  the  flight  crews  to  inform  the  controller  before  executing  an  avoidance  maneuver.  The  pilots 
did  typically  inform  the  controllers  of  their  intent.  Although  notification  was  not  always  prior  to 
maneuvering,  it  was  usually  within  1  minute  of  the  beginning  of  the  maneuver.  By  contrast,  in 
the  shared-separation  condition  where  intent  notification  was  voluntary  and  controllers  couldn’t 
cancel  free  flight  (SSL2),  the  pilot  participants  rarely  notified  the  controller  of  their  intended 
actions.  This  was  likely  why  there  was  less  total  duration  of  air^ground  communications  in 
SS:L2.  The  lack  of  intent  relay  by  the  pilot  participants  was  an  interesting  result  because  the 
findings  of  Corker  et  al.  (2001)  indicate  the  perceived  importance  of  direct  relay  of  intent  to 
controllers  in  a  free  flight  environment. 

Pilot  participant  ratings  of  the  air<->ground  communications  revealed  that  although  the  duration 
of  communications  were  different  for  some  conditions,  they  still  had  adequate  time  for 
air<->ground  communications  in  all  conditions. 

Missed  communications  for  the  four  conditions  did  not  appear  to  indicate  any  clear  finding  since 
the  percentage  of  missed  communications  relative  to  total  communications  was  5%  or  less. 
These  percentages  are  typical  of  the  number  of  missed  communications  in  the  current  en  route 
operational  environment  (Cardosi,  1993).  Although  SS:L1  has  the  highest  percentage  at  5%,  the 
missed  communications  in  that  condition  were  primarily  from  a  single  controller  and  flight  crew 
pair. 
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4.3.2  Information  Requirements  and  Procedures 

4.3.2. 1  Cancellation  of  Free  Flight  Operations 

By  study  design,  pilots  could  cancel  free  flight  in  SS:L1  and  SS:L2  conditions,  and  controllers 
could  cancel  free  flight  in  SS:L1  only.  Each  condition  had  a  total  of  48  conflicts  (3  runs  of  16). 
Of  the  48  conflicts,  39  involved  WJHTC  simulation  pilots  and  9  involved  the  participant  pilots 
and  NASA  simulation  pilot. 

Pilot  participants  did  not  cancel  free  flight  during  SS:L1  or  SS:L2  (nor  were  there  any  scripted 
cancellations  by  simulation  pilots).  Other  investigations  have  found  similar  findings  related  to 
the  reluctance  of  flight  crews  in  shared-separation  to  return  separation  authority  to  the  controller 
(Lozito  et  al.,  2000).  The  pilots  seemed  to  feel  that  with  the  new  technologies,  they  were  able  to 
maneuver  safely  to  resolve  conflicts. 

Controllers  cancelled  free  flight  in  17  of  the  48  conflicts  (35%)  in  SS:L1.  Of  the  9  SS:L1 
conflicts  involving  participant  pilots,  controllers  cancelled  free  flight  5  times  (56%).  Of  the  39 
SS:L1  planned  conflicts  involving  aircraft  flown  by  WHJTC  simulation  pilots,  controllers 
cancelled  free  flight  12  times  (31%).  Thus,  it  is  clear  that  the  controllers  were  in  some  way 
uncomfortable  with  the  resolution  strategy  being  used  by  the  flight  crew  participants  and  the 
scripted  resolutions  of  the  simulation  pilots.  These  data  are  not  too  svuprising  given  the  novelty 
of  the  free  flight  concept.  Previous  research  has  also  foimd  high  incidences  of  controller 
cancellation  in  free  flight  scenarios  (Corker  et  al.,  2001). 

A  review  of  the  circumstances  of  the  participant  pilot  free  flight  cancellations  (5  out  of  9  times) 
indicated  that  the  controllers  seemed  to  cancel  free  flight  when  they  did  not  have  information  of 
at  least  one  of  the  following:  pilot  intent,  air<->air  communications,  and/or  pilots  first  maneuver. 
It  seems  likely  that  a  general  lack  of  these  types  of  information  to  the  controller  contributed  to 
the  discomfort  of  the  controller  in  shared-separation  operations. 

4.3.2.2  Shared-Separation  Procedural  Considerations 

Both  controllers  and  pilot  participants  generally  rated  the  time  available  for  separation  tasks  as 
adequate  or  close  to  adequate.  Based  on  ratings,  the  controllers  seemed  to  have  slightly  more 
time  available  to  assure  aircraft  separation  in  CO:CDTI,  although  this  trend  was  not  very  strong. 
Perhaps  the  information  provided  to  the  flight  crew  through  the  CDTI-AL  gave  the  controller 
more  of  a  perceived  time  buffer  for  conflict  resolution.  Their  direct  control  over  the  conflict 
resolution  is  still  intact,  but  the  flight  crew  is  now  an  extra  set  of  individuals  that  can  closely 
monitor  the  traffic  and  activities  around  conflicts.  The  slightly  lower  ratings  for  SS:L1  and 
SS:L2  are  not  surprising  because  the  controllers  indicated  their  safety  concerns  for  these  two 
conditions. 

The  pilot  participants  did  not  rate  their  time  to  resolve  conflicts  as  different  between  the  two 
shared-separation  conditions  (SS:L1  and  SS:L2).  In  SS:L1,  controllers  cancelled  free  flight  in 
five  of  the  nine  flight  segments.  Evidently,  the  possibility  and  actual  experience  of  controller 
cancellation  did  not  impact  their  sense  of  conflict  resolution  timing. 
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The  controllers  and  pilots  appeared  to  find  that  the  time  allowed  in  the  four  conditions  for 
coordination  and  communication  was  adequate.  This  suggests  the  shared-separation  experienced 
did  not  adversely  affect  the  time  needed  for  coordination  and  communications.  However,  an 
interesting  result  was  the  captains’  ratings  for  CO  as  having  close  to  more  than  adequate  time  for 
coordination  and  communications.  The  flight  crews  did  not  have  access  to  CDTI-AL  or  air4->air 
communications  in  CO.  They  were  not  directly  responsible  for  separation  and,  therefore,  they 
were  not  engaged  in  conflict  detection  and  resolution.  Perhaps  considering  their  ultimate 
responsibilities,  including  fuel  consumption,  the  captains  felt  that  the  controllers  acted  on  the 
conflicts  earlier  than  what  was  necessary  for  this  condition  given  the  few  tasks  required  by  the 
flight  crew.  Although  many  pilots  stated  in  the  debriefing  that  solving  conflicts  earlier  was 
better,  there  was  some  feeling  that  resolutions  may  be  premature  because  the  situation  may 
remedy  itself  by  a  change  in  flight  path  on  the  part  of  one  or  both  of  the  conflicting  aircraft. 

4.3 .2.3  Additional  Information  Requirement  Considerations 

Controller  and  pilot  subjective  comments  indicated  that  URET  and  CDTI-AL  supported  their 
activities  rather  well.  However,  in  some  cases,  controllers  stated  that  the  URET  alerts  were  too 
soon  or  too  late.  Controller  comments  also  strongly  indicated  that  they  would  like  to  have  pilot 
intent  information  earlier.  This  need  was  reinforced  by  the  fact  the  controllers  tended  to  resolve 
conflicts  earlier  than  the  pilots.  If  earlier  intent  information  was  available  to  controllers,  it  is 
possible  that  they  would  not  have  cancelled  free  flight  as  often.  At  present,  CDTI-AL  provides 
alerts  about  7  minutes  prior  to  potential  loss  of  separation,  whereas  URET  provides  conflict 
alerts  about  13-17  minutes  prior  to  potential  loss  of  separation.  Perhaps  having  similar  conflict 
alert  look-ahead  times  would  help  pilots  formulate  and  provide  their  intentions  earlier  for  the 
controller’s  needs.  (However,  the  earlier  the  CDTI-AL  alert,  the  less  certain  the  conflict.)  A 
procedural  solution  may  be  to  require  pilots  to  provide  intentions  to  controllers  at  a  specified 
time  ahead  of  the  conflict  to  help  alleviate  controller  anxiety  about  the  lack  of  timely  intentions. 
On  the  other  hand,  perhaps  more  complicated  traffic  situations  such  as  the  presence  of  multiple 
potential  conflicts  for  the  same  aircraft  that  would  require  pilots  to  execute  the  maneuvers  earlier 
to  avoid  complex  situations  will  force  the  issue,.  One  of  the  controllers  also  suggested  larger 
separation  standard  for  pilots  (i.e.,  10  nm)  so  that  if  free  flight  was  cancelled,  the  controller  will 
have  “buffer”  separation/time  to  get  the  conflict  resolved. 

Both  the  controllers  and  pilot  participants  indicated  that  the  air<->air  frequency  was  useful  in 
SS:L1  and  SS:L2.  However,  it  was  distracting  at  times  for  R-side  controllers  and  therefore  not 
all  R-side  controllers  used  it.  Perhaps  future  technologies,  such  as  datalink,  would  funnel 
communications  to  the  appropriate  sector,  and  the  air<->air  communications  would  not  be  as 
distracting  for  controllers. 

4.3.3  Workload  and  Situation  Awareness 
4.3.3. 1  Workload 

In  general,  based  on  subjective  ratings  and  form  and  debriefing  comments,  controllers  indicated 
that  SS:L1  was  the  most  workload  intensive  and  difficult  condition.  The  reasons  included 
increased  monitoring  tasks,  additional  tasks  to  ensure  that  pilots  were  resolving  conflicts  in  a 
safe  and  timely  manner,  and  planning  multiple  contingency  resolutions  for  conflicts  during 
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SS:L1 .  Controllers  tended  to  prefer  CO  and  CO:CDTI  over  shared-separation  based  on  their 
perception  of  safety  and  workload.  In  general,  all  workload  ratings  were  low,  possibly  indicating 
that  the  shared-separation  tasks  in  this  study  may  not  have  presented  a  significant  challenge  to 
their  overall  workload  in  these  lower  complexity  scenarios.  In  addition,  there  was  no  mix  of  free 
flight  equipped  and  non-free  flight  equipped  aircraft  in  the  scenarios.  Previous  research  (Corker 
et  al.,  2001;  Hilbum  et  al.,  1998)  has  revealed  that  high  traffic  density  and  mixed  equipage  does 
appear  to  negatively  impact  controller  workload. 

Although  always  rated  as  relatively  low*^,  the  pilot  participants  appeared  to  perceive  their 
workload  as  lower  in  the  CO  and  COrCDTI  conditions  compared  to  the  shared-separation 
condition  with  controller  notification  of  intent  and  possible  controller  free  flight  intervention 
(SS:L1).  Interestingly,  similar  to  the  controllers,  the  pilots  also  indicated  that  this  condition  was 
more  workload  intensive  than  the  others.  This  may  be  because  the  pilots  were  required  to 
perform  conflict  detection  and  resolution  activities,  traditionally  controller  activities,  in  addition 
to  their  normal  duties.  They  were  also  required  to  provide  intent  information  imder  SS;L1. 
However,  the  pilots  tended  to  prefer  SS:L2  for  safety,  situation  awareness,  and  workload  reasons 
possibly  because  it  offered  them  complete  autonomy. 

4. 3.3.2  Situation  Awareness 

In  general,  the  controllers  rated  their  situation  awareness  for  the  four  conditions  as  high,  whereas 
the  pilots  appeared  to  believe  that  the  shared-separation  conditions  provided  more  situation 
awareness  than  the  CO  and  CO:CDTI  conditions. 

Pilot  participants  were  provided  with  CDTI-AL  in  COrCDTI,  SS:L1,  and  SS:L2.  In  addition, 
they  were  provided  an  air<->air  fi'equency  for  communications  and  new  procedures  for  separation 
authority  in  SS:L1  and  SS:L2.  This  combination  of  new  technologies  and  responsibilities  may 
explain  why  the  pilot  situation  awareness  was  higher  in  SS:L1  and  SS:L2.  However,  although 
CO:CDTI  offered  more  traffic  information  through  the  CDTI-AL,  their  average  situation 
awareness  was  reported  as  the  same  for  CO  (without  the  CDTI-AL). 

Controllers  were  provided  with  URET  information  for  all  four  conditions.  In  addition,  for  SS:L1 
and  SS:L2,  they  received  monitoring  capabilities  of  the  air<->air  frequency  and  new  procedures 
including  shared-separation  responsibility.  Though  all  ratings  for  situation  awareness  were  high, 
it  is  somewhat  surprising  that  there  was  not  a  noticeable  reduction  in  controller  situation 
awareness  for  SS:L2.  In  SS:L2,  pilots  maneuvered  without  controller  instruction.  Informing  the 
controllers  about  their  maneuvers  was  voluntary,  and  the  pilots  (both  simulation  and  participant 
pilots)  rarely  provided  this  intent.  Therefore,  one  would  expect  overall  controller  situation 
awareness  in  SS;L2  to  be  reduced.  Earlier  studies  have  demonstrated  the  importance  of 
providing  intent  to  controllers  during  shared-separation  tasks  (Corker  et  al.,  2001;  Hilbum, 
Bakker,  Pekela,&  Parasuraman,  1997).  In  addition,  one  would  have  expected  lower  situation 
awareness  in  SS:L2  because  the  procedures  seemed  to  result  in  the  controllers  being  less  directly 


One  reason  for  the  lower  ratings  across  all  conditions  may  be  that  the  en  route  flight  phase  is  typically  considered 
low  workload  for  pilots,  particularly  in  conditions  such  as  those  simulated  where  there  was  no  adverse  weather 
and/or  emergency  events. 
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engaged  in  conflict  management  tasks  (especially  because  no  pilots  actually  cancelled  free  flight, 
and  separation  tasks  were  never  shifted  back  to  the  controllers). 

5.  CONCLUSIONS 


This  study  was  an  initial  attempt  to  examine  the  concept  of  shared-separation  responsibility 
using  high  fidelity  simulators  for  both  ATC  and  flight  deck  operations.  ZME  airspace  was 
simulated,  and  pilots  and  controllers  served  as  participants.  Different  levels  of  shared-separation 
were  examined,  with  each  run  containing  several  conflicting  aircraft.  Pilots  and  controllers  were 
provided  with  display  and  conflict  detection  tools  to  assist  in  these  tasks.  Subjective  and 
objective  data  were  analyzed  to  explore  operational  issues,  provide  recommendations  for 
information  requirements  and  procedures,  and  assess  controller  and  pilot  workload  and  situation 
awareness. 

The  key  operational  issues  identified  were  the  differences  between  controller  and  pilot 
perception  of  safety  and  conflict  resolution  strategies.  There  were  no  losses  of  separation  while 
controllers  or  pilot  participants  were  responsible  for  separation'^.  However,  controllers  felt  that 
safety  was  compromised  in  shared-separation  conditions  when  compared  to  the  current 
operational  conditions.  The  number  of  cancelled  free  flight  operations  in  SS;L1  and  the 
controllers’  indications  that  they  would  have  cancelled  free  flight  in  several  situations  in  SS:L2 
were  evidence  of  their  concern  about  safety  issues.  In  contrast,  the  pilot  participants  rated  both 
shared-separation  conditions  as  being  relatively  safer  than  current  operations.  The  pilots 
indicated  an  overall  preference  for  the  shared-separation  conditions,  particularly  when  they  had 
the  most  flexibility  and  separation  authority  (SS:L2).  Apparently,  the  perceived  flexibility  that 
shared  separation  provided  for  the  pilots  seemed  to  result  in  safety  concerns  and  discomfort  for 
the  controllers.  There  were  also  some  interesting  differences  between  the  pilots  and  controllers 
with  respect  to  conflict  resolution  strategies.  These  included  maneuvering  styles,  timing  of 
action,  and  separation  distances.  For  conflict  resolution,  controllers  preferred  the  use  of  heading 
and  altitude,  whereas  pilots  preferred  heading  and  speed  maneuvers.  Controllers  tended  to 
resolve  conflicts  earlier  and  seemed  to  prefer  greater  separation  distances  than  pilots. 

Both  pilot  and  controller  participants  generally  found  their  tools  for  their  relevant  tasks  to  be 
useful  and  sufficient.  However,  the  notable  issues  related  to  information  requirements  and 
procedures  included  the  availability  of  intent  knowledge,  the  cancellation  of  free  flight,  and  the 
harmonization  of  conflict  detection  tools.  The  lack  of  timely  pilot  intent  knowledge  (as 
perceived  by  controllers)  seemed  to  result  in  free  flight  cancellations  by  controllers.  They  also 
stated  that  this  added  to  their  workload  and  discomfort  with  shared-separation  operations.  If 
earlier  intent  information  was  provided  to  controllers,  it  is  possible  that  they  would  not  have 
cancelled  free  flight  as  often  as  they  did.  Perhaps  having  harmonized  conflict  alert  look-ahead 
times  would  help  pilots  formulate  and  provide  their  intentions  earlier  for  controller  needs.  In 


There  were  two  losses  of  separation  in  SS:L2  runs  (controllers  could  not  cancel  free  flight)  due  to  late 
maneuvering  (script  execution  error)  by  the  WJHTC  simulation  pilots. 
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addition,  both  controllers  and  pilots  found  the  capability  of  air4->air  communications  useful; 
however,  a  more  sophisticated  technology  would  need  to  be  developed. 

Controller  and  pilot  workload  was  generally  low  for  all  conditions,  though  there  were  some 
interesting  observations.  Both  controllers  and  pilot  participants  indicated  that  SS:L1  was  the 
most  workload  intensive  scenario.  Controllers  also  stated  that  it  was  difficult  for  them  and 
suggested  that  their  higher  workload  was  due  to  increased  monitoring,  continuous  assessment  of 
pilot  solutions,  and  the  need  to  plan  contingency  resolutions.  Though  pilots  stated  they  preferred 
shared-separations  operations,  Aey  may  have  rated  workload  higher  in  SS:L1  because,  in 
addition  to  their  normal  duties,  they  had  to  provide  intent  information  and  perform  conflict 
detection  and  resolution  activities.  Controller  situation  awareness  was  consistently  high  for  all 
conditions,  but  pilots  reported  greater  situation  awareness  during  shared-separation  conditions 
compared  to  current  operations.  Their  higher  situation  awareness  may  be  explained  by  the 
combination  of  new  technologies  and  experienced  responsibilities. 

In  this  limited  investigation  of  the  shared-separation  concept,  pilot  participants  tended  to  prefer 
shared-separation  conditions,  whereas  the  controller  participants  tended  to  prefer  current 
operational  conditions.  These  data  were  based  on  a  small  sample  size  and  simplified  scenarios 
and  events.  For  example,  this  simulation  did  not  consider  weather,  climbing  and  descending 
aircraft,  or  other  factors  that  add  to  the  complexity  of  air  traffic  operations.  Future  research 
should  investigate  the  shared-separation  concepts  with  multiple  aircraft  conflicts,  simultaneous 
conflicts,  and  aircraft  in  successive  conflicts.  Finally,  mixed  equipage  operations  in  a  shared- 
separation  context  is  another  characteristic  that  is  likely  to  have  a  strong  impact  on  these 
operations  and  therefore  needs  to  be  investigated.  Although  it  is  premature  to  identify  the  best 
possible  shared-separation  level,  the  results  of  this  study  demonstrate  the  need  to  conduct  further 
research  in  this  area. 
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ACRONYMS 


ACID 

ADO 

ADS-B 

AGIE 

AHP 

ANOVA 

ARC 

ARTCC 

ATC 

CDTI 

CDTI-AL 

CPC 

CO 

CO:CDTI 

CVSRF 

DSR 

EO 

FAA 

FAR 

FL 

FMC 

FMS 

I^F 

IP 

LIT 

M 

MCDU 

MSD 

NAS 

NASA 

OS 

PAS 

PIT 

PVD 

RA 

SAR 

SD 

SEM 

SS:L1 

SS:L2 

SUA 

TA 

TCAS 


Aircraft  Identifier 

Automatic  Datalink  Operator 

Automatic  Dependant  Surveillance-Broadcast 

Air-Ground  Integration  Experiment 

Analytical  Hierarchy  Process 

Analysis  of  Variance 

Ames  Research  Center 

Air  Route  Traffic  Control  Center 

Air  Traffic  Control 

Cockpit  Display  of  Traffic  Information 

Cockpit  Display  of  Traffic  Information  with  Alerting  Logic 

Certified  Professional  Controller 

Current  Operations 

Current  Operations  with  Cockpit  Display  of  Traffic  Information  with 
Alerting  Logic 

Crew  Vehicle  Systems  Research  Facility 
Display  System  Replacement 
Expert  Observers 
Federal  Aviation  Administration 
Federal  Aviation  Regulations 
Flight  Level 

Flight  Management  Computer 
Flight  Management  System 
Integration  and  Interoperability  Facility 
Internet  Protocol 
Little  Rock 
Mean 

Multi-Function  Control  Display  Unit 

Minimum  Separation  Distance 

National  Airspace  System 

National  Aeronautics  and  Space  Administration 

Operations  Supervisor 

Pseudo  Aircraft  Systems 

Push-to-Talk  Transmissions 

Plan  View  Display 

Resolution  Advisory 

System  Analysis  and  Recording 

Standard  Deviation 

Standard  Error  of  the  Mean 

Shared-Separation  Level  1 

Shared-Separation  Level  2 

Special  Use  Airspace 

Traffic  Advisory 

Traffic  Alert  and  Collision 
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TGF 

Target  Generation  Facility 

URET 

User  Request  Evaluation  Tool 

VNTSC 

Volpe  National  Transportation  Systems  Center 

VOR 

Very  High  Frequency  Omni  Directional  Radio  Range 

VORTAC 

VOR  Tactical  Air  Navigational  Aid 

WAK 

Workload  Assessment  Keypad 

WJHTC 

William  J.  Hughes  Technical  Center 

ZME 

Memphis  ARTCC 
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GLOSSARY  OF  TERMS 


URET 

User  Request  Evaluation  Tool.  The  look-ahead  time  for  conflict 
detection  using  flight  plan  on  the  URET  is  approximately  20  minutes. 

CDTI-AL 

Cockpit  Display  of  Traffic  Information  with  Alerting  Logic.  The 
CDTI-AL  typically  alerts  the  flight  crews  approximately  6-7  minutes 
prior  to  the  closet  point  of  approach  between  aircraft. 

Workload 

Workload  is  defined  as  combined  cognitive  and  physical  demands 
experienced  by  an  operator.  The  workload  experienced  by  an  operator 
depends  on  the  task,  skill,  knowledge,  experience,  abilities,  and 
training.  Generally,  workload  is  considered  as  an  operator’s  response 
to  taskload. 

Situation 

Awareness 

Situation  awareness  is  defined  as  an  operator’s  ability  to  integrate 
information  related  to  state  of  a  task,  operation,  equipment,  and 
environment;  make  necessary  predictions;  and  take  the  necessary 
decisions  and  suitable  actions.  Several  other  definitions  of  situation 
awareness  exist. 

Communication 

A  series  of  two  or  more  transmissions  between  a  member  of  the  flight 
crew  and  a  controller  on  a  single  topic;  an  exchange  of  a  message  and 
a  response;  the  duration  of  a  transaction  is  measured  from  the  onset  of 
speech  for  the  initial  message  to  the  end  of  the  response  from  the  other 
party. 

Transmission 

A  verbal  message  from  the  controller  to  the  pilot  or  vice  versa;  the 
duration  of  which  is  measured  from  the  onset  of  speech  to  the  end  of 
the  message  (offset  of  speech). 

CO 

This  condition  emulates  today’s  ATC  environment;  that  is,  the 
controller  is  responsible  for  separation  assurance  of  all  aircraft. 

URET  is  operational. 

CO:CDTI 

This  condition  emulates  today’s  ATC  environment.  URET  is 
operational.  In  this  condition,  however,  the  flight  crews  have 

CDTI-AL  available  to  them. 

SS;L1 

This  condition  emulates  a  subset  of  the  RTCA  definition  of  the  free 
flight  environment.  URET  and  CDTI-AL  are  operational.  Controllers 
have  specific  procedures  for  coordinating  URET  alerts.  Flight  crews 
are  free  to  initiate  any  maneuver  (i.e.,  change  heading,  altitude,  or 
speed),  provided  they  first  inform  ATC  of  their  intentions.  Controllers 
and  flight  crews  both  could  cancel  free  flight. 

SS;L2 

This  condition  incorporates  all  the  conditions  of  SS:L1  with  some 
modifications.  Flight  crews  are  free  to  maneuver,  however  they  are 
not  required  to  inform  ATC  of  their  intentions.  Flight  crews  may 
cancel  free  flight,  but  controllers  can  not. 
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Appendix  A 


Controller  Daily  Schedule 


Daily  Schedule  for  Controllers 


Time  (EST) 

Day  1 

Day  2 

Day  3 

Day  4 

Days 

12:00pni 

Review 

:15 

Controller 

Review 

:30 

:45 

Briefing 

01:00pm 

:15 

Training 

Run  3 

:30 

:45 

ATC 

02;00pm 

:15 

Laboratory 

Familiarization 

Forms  & 
Discussion 

:30 

:45 

MEAL 

03:00pm 

:15 

MEAL 

BREAK 

BREAK 

:30 

:45 

Participant 

MEAL 

Participant 

04:00pm 

:15 

Travel 

BREAK 

Run  1 

Run  4 

Travel 

:30 

:45 

Training  Run  1 

05:00pm 

Forms  & 

:15 

Discussion 

Forms  & 

Discussion 

:30 

BREAK 

Discussion 

BREAK 

:45 

BREAK 

06:00pm 

:15 

Training  Run  2 

Debriefing 

:30 

Discussion 

Run  2 

:45 

BREAK 

07:00pm 

:15 

Training  Run  3 

Buffer 

Buffer 

:30 

Forms  & 

Buffer 

:45 

Discussion 

Discussion 

Buffer 
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FAA  William  J.  Hughes  Technical  Center 
Points  of  Contact 

Karen  DiMeo,  ACT-540 

Dr.  Randy  Sollenberger,  ACT-530 
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Controller  Briefing 
Air-Ground  Integration  Experiment 


1  Introduction 

Over  the  last  decade,  the  recognized  need  for  more  efficient  Air  Traffic  Control  (ATC) 
has  produced  several  new  automation  systems.  The  Federal  Aviation  Administration 
(FAA)  has  implemented  programs  such  as  the  Enhanced  Traffic  Management  System 
(ETMS)  and  the  Center  TRACON  Automation  System  (CTAS)  to  assist  air  traffic 
controllers  and  managers  with  their  duties.  For  the  cockpit,  automated  Flight 
Management  Systems  have  been  developed  to  help  flight  crews  plan  and  execute  fuel- 
efficient  routings  to  destination  airports.  Systems  such  as  these  are  intended  to  provide 
means  for  more  efficient  use  of  the  National  Airspace  System  (NAS).  However,  the  fact 
remains  that  the  tools  in  the  cockpit  and  on  the  groimd  are  not  integrated  thereby  limiting 
the  maximum  benefit  obtained  from  these  technologies.  In  the  spirit  of  progress,  the 
FAA  continues  to  address  such  problems  by  testing  and  implementing  new  technologies 
and  procedures,  and  by  investigating  future  concepts. 

‘Free  Flight  ’  as  described  by  the  RTCA  Task  Force  3,  sets  forth  a  concept  of  future  air 
traffic  management  that  requires  a  tightly  integrated  system  to  meet  the  freer  and  more 
collaborative  nature  of  air  traffic  management  in  the  fiiture.  While  aircraff-to-aircraft 
separation  will  remain  the  responsibility  of  service  providers,  and  in  most  cases,  will 
remain  solely  their  responsibility,  today’s  practice  of  visual  separation  by  pilots  in 
terminal  areas  is  expanded  to  allow  all-weather  pilot  separation  when  feasible.  The 
increased  use  of  shared-separation  responsibility  will  be  possible  through  the  use  of 
traffic  displays  on  the  flight  deck,  as  well  as  rules,  procedures,  and  training  programs  that 
modify  the  roles  and  responsibilities  of  users  and  service  providers.  Human  factors 
analyses  and  human-in-the-loop  simulations  will  help  determine  the  appropriate 
allocation  of  tasks  between  service  providers,  users,  and  automation  systems. 

In  response,  the  National  Aeronautics  and  Space  Administration  (NASA)  Ames  Research 
Center  (ARC)  has  developed  a  Cockpit  Display  of  Traffic  Information  (CDTI)  prototype. 
The  CDTI  includes  embedded  conflict  alerting  logic  (AL)  that  predicts  the  probability  of 
an  encounter  with  another  aircraft.  The  CDTI-AL  relies  on  Automated  Dependant 
Surveillance  (ADS)  technology  to  supply  the  position  and  trajectory  information  of  all  air 
traffic  in  the  vicinity.  This  prototype  ‘decision  support  tool’  is  intended  to  enhance  flight 
crews’  situation  awareness  and  provide  them  more  autonomy  in  the  NAS.  In  addition, 
the  MITRE  Corporation  has  developed  a  groxmd-based  conflict  probe  and  trial-planning 
tool  for  use  by  air  traffic  controllers.  This  prototype  decision  support  tool,  entitled  User 
Request  and  Evaluation  Tool  (URET),  is  currently  being  evaluated  at  the  Indianapolis 
and  Memphis  Air  Route  Traffic  Control  Centers.  While  there  have  been  studies  done  on 
each  of  these  tools  individually,  there  is  a  need  to  investigate  how  they  might  work 
together  in  a  shared-separation  environment. 
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An  Experiment  Working  Group  (EWG)  has  been  formed  to  investigate  the  effects  of 
shared-separation  authority  on  flight  operations  when  both  the  air  and  ground  have 
enhanced  traffic  and  conflict  alerting  systems.  The  EWG  is  represented  by  FAA  Head 
Quarters,  the  FAA  William  J.  Hughes  Technical  Center  (WJHTC),  NASA  ARC,  and  the 
Volpe  National  Transportation  Systems  Center  (VNTSC).  The  study  is  co-sponsored  by 
the  FAA  (AAR-100,  ASD-130,  ATP-400)  and  NASA  (Advanced  Air  Transportation 
Technologies  Program). 

AGIE  is  the  first  high  fidelity,  real-time,  human-in-the-loop  simulation  in  a  planned 
series  of  studies  to  investigate  shared-separation  operations.  More  studies  will  be 
conducted  in  the  future  prior  to  making  any  conclusive  recommendations  about  shared- 
separation  responsibility. 

2  Objectives 

This  experiment  is  intended  to  provide  an  initial  examination  of  the  effect  of  shared- 
separation  authority  on  flight  operations  when  both  air  and  ground  have  enhanced  traffic 
and  conflict  alerting  systems.  There  will  be  strong  emphasis  on  identifying  and 
evaluating  the  human  factors  impact.  Under  the  conditions  simulated,  the  specific 
objectives  are: 

•  Identify  operational  issues  (e.g.,  communications,  procedures,  etc.)  that  affect  shared- 
separation  operations. 

•  Provide  recommendations  for  the  information  requirements  and  procedures  necessary 
to  facilitate  shared-separation  operations. 

•  Evaluate  the  effect  of  shifting  separation  authority  on  controller  and  pilot  workload 
and  situation  awareness. 

3  Definitions  and  Terms 

The  following  provides  description  of  definitions  and  terms  that  are  essential  in  this 
experiment. 

•  URET-  User  Request  Evaluation  Tool. 

The  look  ahead  time  for  conflict  detection  using  flight  plan  on  the  URET  is 
approximately  20  minutes.  You  will  be  required  to  coordinate  RED  alerts  with  adjacent 
sectors.  The  URET  will  be  operational  under  all  scenarios. 

•  CDTl-AL-  Cockpit  Display  of  Traffic  Information  with  Alert  Logic. 

The  CDTI-AL  typically  alerts  the  flight  crews  approximately  6-7  minutes  prior  to  the 
closet  point  of  approach  between  aircraft.  Therefore,  in  most  cases  URET  will  provide 
conflict  alerts  prior  to  the  CDTI-AL.  See  Appendix  A  for  detailed  description. 

•  ifF-  Integration  and  Interoperability  Facility  (Building  27) 

•  RDHFL-  Research  Development  and  Human  Factors  Laboratory  (Building  28) 
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•  WAK-  Workload  Assessment  Keypad 


4  Scenario  Descriptions 

Following  training,  you  will  be  presented  four  scenario  types  in  a  random  order.  This 
section  contains  a  description  of  each  scenario,  conflict  resolution  and  coordination 
information,  phraseology  conditions,  and  your  role  while  participating. 

4.1  T raining  scenarios 

There  will  be  three  training  scenarios,  one  each  for  CO:CDTI,  SS:L1,  and  SS:L2 
operations.  See  below  for  descriptions  of  scenario  characteristics. 

4.2  Baseline  (CO)  scenario  characteristics 

This  control  condition  will  simulate  today’s  ATC  environment.  URET  will  be 
operational.  Standard  air  traffic  procedures  defined  in  the  Controller  Handbook  71 10.65, 
Federal  Aviation  Regulations  (FAR)  Part  91 ,  and  the  Aeronautical  Information  Manual 
will  apply. 

•  Today’s  air  traffic  control  operation  where  controllers  have  separation  responsibility. 

•  Pilots  do  not  have  access  to  CDTI-AL  in  the  cockpit. 

•  Controllers  have  access  to  URET. 

•  Pilots  cannot  initiate  a  maneuver  without  ATC  clearance. 

4.3  CO:CDTI  scenario  characteristics 

This  control  condition  will  simulate  today’s  ATC  environment  with  the  addition  that  the 
flight  crews  will  have  access  to  CDTI-AL.  URET  will  be  operational.  Standard  air 
traffic  procedures  defined  in  the  Controller  Handbook  71 10.65,  FAR  Part  91,  and  the 
Aeronautical  Information  Manual  will  apply.  Pilots  may  request  alternate  routes  to 
maximize  fuel  efficiency  or  when  potential  conflicts  are  detected  with  the  CDTI-AL,  but 
controllers  retain  the  authority  to  approve/deny  all  pilot  requests. 


•  Controllers  will  take  the  initiative  to  resolve  conflicts  between  aircraft. 

•  Pilots  do  have  access  to  CDTI-AL  in  the  cockpit. 

•  Controllers  have  access  to  URET. 

•  Pilots  cannot  take  any  action  (other  than  emergency  or  TCAS  RA)  without  clearance 
from  ATC. 

•  Pilots  can  query  controllers  (e.g.  about  potential  conflicts  or  traffic)  and  make 
requests  based  on  information  from  their  CDTI-AL  displays. 

4.4  SS:L1  scenario  characteristics 

This  control  condition  will  emulate  a  subset  of  the  RTCA  definition  of  the  Free  Flight 
environment.  URET  and  CDTI-AL  will  be  operational.  The  initial  flight  plan  and 
altitude  will  be  considered  as  optimum  for  the  current  conditions.  Standard  air  traffic 
rules  defined  in  the  Controller  Handbook  71 10.65,  FAR  Part  91  and  the  Aeronautical 
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Information  Manual  will  still  apply,  with  the  major  exception  that  flight  crews  will 

initially  provide  their  own  separation.  However,  pr/or  to  maneuvering,  they  must  first 

inform  ATC  of  their  intended  actions.  Controllers  can  cancel  Free  Flight. 

•  Shared-separation  operation  and  responsibility. 

•  Pilots  do  have  access  to  CDTI-AL  in  the  cockpit. 

•  Controllers  have  access  to  URET. 

•  Flight  crews  are  free  to  maneuver  in  any  direction  including  vertically  provided  they 
first  inform  ATC. 

•  An  air-to-air  frequency  is  available.  Controllers  can  monitor  the  air-to-air  frequency 
as  desired,  but  it  is  not  required. 

•  Standard  separation  rules  of  5  miles  laterally  or  1000/2000  ft.  vertically  shall  be 
observed  by  ATC  and  flight  crews. 

•  To  issue  a  control  instruction  to  a  pilot,  controllers  must  CANCEL  their  Free  Flight 
operation. 

•  Controllers  may  only  CANCEL  FREE  FLIGHT  {for  one  or  a  pair  of  aircraft,  sector¬ 
wide  cancellation  is  NOT  allowed)  if  they  have  queried  at  least  one  of  the  subject 
aircraft  as  to  pilot’s  intentions. 

•  Aircraft  whose  free  flight  has  been  canceled  will  remain  under  ATC  control 
unless/until  the  controller  RESUMES  Free  Flight. 

•  Controllers  shall  update  the  Host/DSR  database  when  flight  plans  are  changed. 

•  Controllers  shall  issue  Traffic  Alerts  using  prescribed  phraseology  to  the  aircraft 
involved  in  a  RED  URET  alert. 

•  Controllers  shall  coordinate  all  RED  URET  alerts  on  aircraft  not  under  their  control 
with  the  controlling  sector  using  the  prescribed  phraseology. 

•  Controllers  receiving  a  coordinated  Traffic  Alert  shall  forward  this  to  the  subject 
aircraft  unless  that  aircraft  has  already  advised  that  a  resolution  is  in  progress. 

•  Controllers  shall  have  the  prerogative  to  wait  to  issue  a  Traffic  Alert  until  the  subject 
aircraft  is  under  his/her  control. 

•  Controllers  shall  coordinate  any  aircraft  action  that  will  affect  another  controller’s 
airspace. 

•  Flight  crews  can  cancel  Free  Flight  for  their  aircraft  and  request  ATC  to  intervene  at 
any  time. 

4.5  SS:L2  scenario  characteristics 

This  control  condition  incorporates  all  the  conditions  of  SS:L1  with  the  following 

changes: 

•  Flight  crews  are  not  required  to  inform  the  controller  before  initiating  any  maneuver. 

•  Controllers  are  not  required  to  issue  Traffic  Alerts  to  aircraft,  but  may  do  so. 

•  Controllers  must  still  coordinate  all  RED  URET  alerts  on  aircraft  not  under  their 
control  with  the  controlling  sectors. 

•  Controllers  may  NOT  CANCEL  FREE  FLIGHT;  however,  conflict  detection  and 
resolution  measures  will  be  collected. 


•  Flight  crews  can  cancel  Free  Flight  for  their  aircraft  and  request  ATC  to  intervene  at 
any  time. 

4.6  Conflict  resolution  rules 

During  SS:L1  and  SS:L2  scenarios,  pilots  are  instructed  to  use  FAR  Part  91  right-of-way 
rules  (when  possible)  while  resolving  conflicts.  The  pilot  right-of-way  rules  are  as 
follows: 

•  The  aircraft  on  the  others  right  has  the  right-of-the-way. 

•  The  aircraft  being  overtaken  has  the  right-of-the-way. 

•  The  aircraft  that  are  head-on  should  each  alter  course  to  the  right. 

•  During  most  conflict  situations,  the  aircraft  that  does  not  have  right-of-way  will 
initiate  the  coordination  with  the  aircraft  that  has  right  of  way. 

4.7  Coordination  of  URET  red  alerts 

You  will  be  required  to  coordinate  RED  URET  alerts  with  other  sectors  under  all 
scenario  types.  In  other  words,  you  will  coordinate  all  RED  alerts  for  those  aircraft 
where  the  conflict  is  predicted  to  occur  in  your  sector,  but  the  aircraft  is  not  within  your 
sector  and/or  control.  However  you  need  not  coordinate  RED  alerts  with  “brown”  URET 
aircraft  ID’s  or  those  with  “UNK”  as  the  sector.  For  simulation  purposes,  these  are 
considered  “nuisance”  RED  alerts. 

4.8  Phraseology 

Controllers  will  use  standard  phraseology  as  described  in  the  FAA  Order  71 10.65  to 
minimize  the  possibility  of  misunderstandings  among  other  controllers  and  pilots.  In 
addition,  the  following  phraseology  will  be  used  in  this  simulation: 

4.8.1  Simulation  Phraseology 

•  Cancellation  of  Free  Flight  (SS:L1  only). 

Controller  -  '’’'ACID  (and  ACID),  Free  Flight  canceled"  and  issue  the  appropriate 
control  action. 

•  Resumption  of  Free  Flight  (SS:L1  only). 

Controller  -  '’’’ACID  (and  ACID),  resume  Free  Flight." 

•  Acknowledge  Pilot  intentions  (SS:L1  only). 

Pilot  -  informs  controller  of  an  intended  maneuver. 

Controller  -  "ACID,  roger.  ” 

•  Aircraft  coordination  for  RED  URET  alerts  (SS:L1  and  SS:L2). 

Controller  -  '"''ACID,  traffic  alert  with  ACID  at  (altitude)  at  (time),  advise  intentions  ” 


B-6 


Sector  coordination  for  RED  URET  alerts  (SS:L1  and  SS:L2). 

Controller  Sector  1  -  “Traffic  alert  ACID” 

Controller  Sector  2  -  “Go  ahead' 

Controller  Sector  1  -  “ACID  with  ACID  at  altitude  at  time” 

Controllers  may  continue  to  use  their  operating  initials  at  the  end  of  coimnunications  as 
usual. 

Pilots  will  be  trained  on  simulation  phraseology  for  their  communications  on  the  air-to- 
air  frequency. 

Controllers  will  be  given  an  opportimity  to  comment  on  all  phraseology  at  the  end  of  this 
experiment. 

The  following  table  provides  a  summary  of  scenario  characteristics. 

Table  1.  Scenario  Description  Summary 


Characteristics 

CO 

COtCDTI 

SS:L1 

SS:U 

URET  available 

. >r' 

V 

V 

V 

Need  to  coordinate  URET  red  alerts  with  other  sectors 

V 

V 

V 

V 

Controllers  have  full  separation  responsibility 

V 

V 

Pilots  must  request  a  clearance  from  controllers  prior  to  flight 
plan  changes 

■■ 

V 

CDTI-AL  available 

V 

V 

V 

Air-to-air  frequency  available 

V 

V 

Pilots  use  right-of-way  rules  while  resolving  potential  conflicts 

V 

Pilots  and  controllers  have  shared-separation  responsibility  - 
Pilots  can  cancel  free  flight 

■■ 

■■ 

New  controller  phraseology 

bhh 

bihhhi 

V 

Pilots  must  inform  controllers  prior  to  flight  plan  changes 

HEIH 

H 

Pilots  and  controllers  have  shared-separation  responsibility  - 
Controllers  can  cancel  Free  Flight 

■■ 

Pilots  and  controllers  have  shared-separation  responsibility  - 
Controllers  will  report  when  the  would  cancel  Free  Flight,  but 
can  not  actually  cancel 

V 

Pilots  do  not  have  to  inform  controller  prior  to  flight  plan 
changes 

■■ 

5  Experiment  Procedure 

You  will  be  participating  in  this  study  for  a  period  of  3  days  (from  12  to  8  p.m.).  On  the 
first  day,  there  will  be  an  experiment  briefing  and  training.  The  experiment  briefing  will 
provide  an  overview  of  the  experiment,  objectives,  and  your  role.  The  training  will 
provide  you  with  laboratory  familiarization  and  an  opportunity  to  control  air  traffic  under 
the  different  control  conditions  of  the  experiment.  On  the  second  day,  you  will  be 
provided  additional  training  as  necessary.  Following  the  training,  data  collection  runs 
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will  begin.  The  data  collection  runs  will  continue  through  the  third  day.  At  the  end  of  all 
data  collection  runs,  there  will  be  an  unstructured,  group  debriefing  session.  The  purpose 
of  this  debriefing  is  to  provide  you  an  opportunity  to  share  information  that  is  not 
captured  in  the  forms. 

On  the  first  day,  you  will  be  given  a  consent  form  and  a  background  data  collection  form. 
After  completing  these  forms,  you  will  participate  in  three  training  runs,  each 
corresponding  to  CO:CDTI,  SS:L1,  and  SS:L2.  This  will  conclude  the  first  day.  At  the 
beginning  of  the  second  day,  you  will  be  given  more  opportunity  for  training.  Following 
the  training  runs,  the  data  collection  runs  will  begin. 

At  the  beginning  of  each  run,  a  researcher  will  inform  you  of  the  control  condition  (e.g., 
CO:CDTI,  SS:L1,  SS:L2,  and  CO).  You  will  be  provided  with  an  aid  at  the  control 
position  describing  rules  of  the  operation.  You  will  also  be  given  a  sector  briefing  as  you 
join  the  run.  Except  for  training  scenarios,  the  duration  of  each  scenario  is  95  minutes. 
The  training  scenarios  are  45  minutes  long. 

There  will  be  two  Expert  Observers  in  the  control  room.  One  observer  will  be  assigned 
to  sector  44  and  the  other  will  be  assigned  to  sector  21 .  During  each  scenario,  the 
observers  will  watch  sector  operations  and  record  interesting  and  critical  events.  The 
observers  will  be  either  a  current  controller  or  a  current  supervisor. 

Participation  in  this  study  is  strictly  voluntary  and  the  privacy  of  participants  will  be 
protected.  No  individual  names  or  identities  will  be  recorded  or  released  in  any  reports. 
Therefore,  you  will  be  assigned  a  participant  code  (e.g.,  C 1 ,  C2,  C3,  eind  C4)  that  will 
remain  the  same  throughout  the  experiment.  Strict  adherence  to  all  federal,  union,  and 
ethical  guidelines  will  be  maintained  throughout  the  study. 

You  will  be  assigned  a  specific  R  or  D-side  position  for  sector  21  or  sector  44.  Your 
sector  and  position  assignment  will  remain  the  same  throughout  the  experiment. 

Appendix  A  provides  general  maps  and  layouts  of  the  buildings  where  the  simulation 
experiment  will  be  conducted. 

5.1  Simulation  Environment 

1 .  The  simulation  pilots  will  emulate  real  world,  current,  and  qualified  pilots.  Most  of 
the  simulation  pilots  have  airline  or  general  aviation  pilot  experience.  You  will  be 
given  an  opportunity  to  visit  the  pilot  workstation  laboratories. 

2.  VSCS  is  not  available.  However,  an  alternate  communications  system  will  emulate 
similar  functionality  for  the  air-to-air,  air-to-ground,  and  ground-to-ground 
communications. 

3.  You  will  be  trained  on  the  communications  system. 

4.  The  URET  version  that  is  used  in  this  experiment  is  the  same  that  is  currently 
operational  at  the  Memphis  Center.  URET  capabilities  are  identical. 
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5.  The  DSR  version  that  is  used  in  this  experiment  is  BAB03.  The  Memphis  Center 
uses  BAB04  version.  The  differences  between  these  versions  are  marginal  and  not 
significant  for  this  experiment.  You  will  be  trained  on  the  differences. 

6.  There  are  no  ascending  and/or  descending  aircraft  in  the  traffic  scenarios.  The  reason 
for  their  exclusion  was  for  complexity  issues  related  to  experimental  design. 

7.  Some  aircraft  targets  may  unnaturally  and  unexpectedly  appear  in  handoff  status  very 
close  to  your  sector  boundary.  This  is  a  limitation  of  the  simulation  environment. 

S.  You  need  to  disable  the  auto  hand-off  function  by  entering  the  QA  F  command  on 
DSR.  The  expert  observers  will  assist  you  as  required. 

5.2  Airspace  Description 

1 .  The  airspace  of  ZME  sectors  2 1  and  20  are  combined  for  the  purposes  of  this 
experiment.  We  recognize  that  they  may  not  be  combined  for  similar  air  traffic 
complexity  in  the  real  environment. 

2.  The  combined  sector  altitudes  include  FL  240  and  above.  For  the  purposes  of  the 
experiment,  the  combined  sector  will  be  referred  to  as  sector  21. 

3.  Sector  44  altitudes  include  FL240  and  above. 

4.  In  addition  to  sectors  21  and  44,  there  will  be  a  sector  78.  Sector  78  emulates  a 
combination  of  all  other  sectors  that  are  adjacent  to  sector  21  and  sector  44. 
Therefore,  if  you  need  to  communicate  with  a  sector  controller  that  is  not  from  sector 
21  and/or  44,  you  need  to  contact  sector  78. 
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Figure  I.  Memphis  ARTCC  sectors  21  and  44 
5.3  Frequencies 

In  addition  to  Sector  21  and  44  frequencies,  the  experiment  emulates  an  air-to-air 
frequency.  The  air-to-air  frequency  is  only  available  during  SS:L1  and  SS:L2  conditions. 
On  the  air-to-air  frequency,  the  pilots  will  be  able  to  communicate  among  themselves. 
Controllers  will  be  able  to  selectively  monitor  the  frequency  as  desired  but  are  not 
permitted  to  transmit  on  the  frequency  for  ANY  reason.  Table  2  provides  the  frequencies 
used  in  this  experiment. 
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Table  2.  Sector  frequencies 


Type 

Frequency 

Sector  21 

132.42 

Sector  44 

124.92 

Air-to-air 

122.75 

Sector  78 

123.45 

6  Data  Collection  Requirements  and  Methods 

Participants  and  observer  data  will  be  kept  locked  and  secured.  No  individual  names  or 
identities  will  be  recorded  or  released  in  any  reports. 

6.1  Forms  and  Questionnaires 

The  following  table  provides  a  summary  of  all  data  collection  forms  and  questionnaires 
that  will  be  completed  by  expert  observers  and/or  participant  controllers. 

Table  3.  Forms  for  Expert  Observers  and  Participant  Controllers 


Forms 

Purpose 

Frequency 

Consent  Form 

Consent  to  voluntary  participation  in  the 
study.  Understanding  of  participant  rights. 

Once  prior  to 
simulation 

Controllers  & 
Observers 

Background 
Information  Form 

Provide  demographic  information  related 
to  professional  experience 

Once  prior  to 
runs 

Controllers  & 
Observers 

During-the-Run 

Form 

Record  critical  and  interesting  events, 
controller  actions,  and  observations  of  the 
impact  of  shared-separation  operations. 

Every  run 

Observers  only 

Post-Run  Form 

Record  information  concerning  overall 
workload,  situation  awareness,  the  impact 
of  conflicts,  and  shared-separation 
operations. 

End  of  every 
run 

Controllers  & 
Observers 

Post-Simulation 

Form 

Respond  to  general  questions  about  shared- 
separation  operations  and  the  overall 
experiment. 

Once  at  the 
end  of 
simulation 

Controllers  & 
Observers 

6.2  Workload  Assessment  Keypad  (WAK)  Ratings 

You  will  be  asked  to  record  a  measure  of  your  overall  workload  every  5  minutes  on  the 
WAK.  The  WAK  will  illuminate  and  emit  a  low-level  beep  every  5-minutes.  At  those 
cues,  press  the  key  that  corresponds  to  your  workload  level. 
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While  reporting  your  workload  rating,  please  consider  combined  mental  and  physical 
workload.  Your  rating  should  reflect  an  instantaneous  measure.  In  other  words,  rate 
your  overall  workload  at  that  moment,  not  as  a  cumulative  measure  of  the  last  5  minutes. 
The  workload  rating  scale  is  shown  in  Table  4. 

Table  4.  Workload  Rating  Scale 


1 

2 

3 

4 

■■■■ 

Very  Low 

Moderate 

6.3  Additional  points/events  of  interest 

In  addition  to  the  5 -minute  workload  ratings,  you  will  be  asked  to  identify  and  report 

additional  points  during  the  scenarios.  These  points  are  defined  as  follows: 

•  Point  A  -  When  you  detect  a  potential  conflict  between  a  pair  of  aircraft  with  a 
reasonable  certainty.  (When  you  begin  to  pay  more  attention  to  a  pair  of  aircraft  due 
to  the  possibility  of  conflict).  Identify  for  all  scenarios. 

•  Point  B  -  When  you  would  take  action  to  resolve  a  potential  conflict  under  current 
operating  rules  (as  described  in  71 10.651.  Assume  that  pilots  do  not  have  CDTI-AL 
in  their  cockpits.  Identify  for  SS:L1  and  SS:L2  only. 

•  Point  C  -  When  you  would  take  an  action  to  resolve  a  potential  conflict  under  free 
flight  conditions  where  pilots  are  self-separating  themselves.  Assume  that  pilots  have 
CDTI-AL  in  their  cockpits.  Identify  for  SS:L2  only. 


Table  5.  WAK  and  Additional  Data  Collection  Summary 


Scenario 

WAK  (every  5 
minutes) 

Point  A 

Point  B 

Point  C 

CO 

V 

V 

CO:CDTI 

V 

V 

SS:L1 

V 

V 

V 

SS:L2 

V 

^  , 

V 

V 

The  controller  (either  R  or  D)  who  first  identifies  a  point  of  interest  (Point  A,  B,  or  C) 
will  inform  the  expert  observer.  The  same  controller  will  point  out  the  aircraft  pair  to  the 
expert  observer.  The  expert  observer  will  then  document  the  aircraft  pair,  simulation 
time,  and  the  point  of  interest.  The  observer  will  also  record  if  point  A  was  identified 
using  URET. 
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Appendix  C 

Controller  and  Expert  Observer  Forms 


Air-Ground  Integration  Experiment  (AGIE) 

Background  Information  Form 

This  form  is  to  be  completed  by  all  AGIE  Controllerand  Expert  Observer 
participants.  This  form  requests  general  background  information. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and 
to  encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Controller 
or  Observer  A,  B,  C,  etc.). 
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Air-Ground  Integration  Experiment  (AGIE) 
Background  Information  Form 

Controller  or  Observer  (circle  one)  jD#. 

Date: _ ! _ / _ 

Male  or  Female  (circle  one) 


1 .  What  is  your  age? 

Years: _ 

2.  What  is  your  total  experience  as  an  air  traffic  control  specialist  (include  FAA  and  military 
experience)? 

Years: _  Months: _ 

3.  How  long  have  you  actively  controlled  traffic  for  the  FAA? 

Years: _  Months: _ 

4.  How  long  have  you  been  a  Full  Performance  Level  (FPL)  controller? 

Years: _  Months: _ 

5.  How  long  have  you  been  a  controller  at  Memphis? 

Years: _  Months: _ 

6.  How  long  have  you  been  using  DSR? 

Years: _  Months: _ or  Not  Trained 

7.  How  long  have  you  been  using  URET? 

Years: _  Months: _ or  Not  Trained 

8.  What  is  the  percentage  of  time  you  use  URET? 

Sector  2 1 :  Sector  44 : _  Overall : _ 
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Air-Ground  Integration  Experiment  (AGIE) 

Observer  Post-Run  Form 

This  form  is  to  be  completed  by  observers  participating  in  AGIE  and  requests  information 
regarding  your  overall  observations  and  judgments  during  the  run. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and  to  encourage 
unbiased  reporting.  Findings  will  be  reported  generically  (Observer  A,  B,  C,  etc.). 

You  are  encouraged  to  write  any  additional  comments  that  you  feel  are  important. 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are  encouraged  to  write 
any  additional  comments  you  feel  are  important. 
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Air-Ground  Integration  Experiment  (AGIE) 
Observer  Post-Run  Form 


Observer  ID#: _  Controller  IDs,  R-Side: _  D-Side: 

Scenario: _ 

Run#: _  Sector: _ 

Date: _ / _ ! _ 

Where  applicable,  please  circle  the  most  accurate  response. 


The  term  physical  taskload  refers  to  the  physical  activities  associated  with  accomplishing  tasks. 
For  example,  performing  the  physical  actions  associated  with  entering  keyboard  data, 
communications,  manipulating  the  trackball  are  components  of  physical  taskload. _ 


1 .  Circle  the  response  that  describes  the  controller  physical  taskload  during  the  simulation  run. 


R-Controller 

1 

Very  Low 

2 

3 

Moderate 

4 

5 

Very  High 

D-Controller 

1 

Very  Low 

2 

3 

Moderate 

4 

5 

Very  High 

Comments. 


2.  Identify  factors  that  interfered  with  maintaining  separation. 


3.  Identify  any  aircraft  separation  issues  that  became  apparent, 


4.  Identify  any  ground-to-ground  coordination  /  communications  issues  that  became  apparent. 


5.  Identify  any  coordination  /  communications  issues  between  R-side  and  D-side  that  became 
apparent. 


6.  Identify  any  ground-to-air  communications  issues  that  became  apparent. 


7.  In  your  opinion,  how  much  vertical  and  horizontal  separation  represents  effective  safety 
margin? 

Vertical  separation  (below  FL290): _ _ (ft.) 

Vertical  separation  (above  FL290): _ (ft.) 

Horizontal  separation: _ (nm) 
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8.  Comment  on  any  other  issues  that  you  observed  during  this  run  that  could  help  the 
researchers  understand  the  events  as  they  occurred. 


Air-Ground  Integration  Experiment  (AGIE) 

Controller  Exit  Form 

This  is  the  final  form  to  be  completed  by  air  traffic  controllers  participating  in 
AGIE  and  requests  information  regarding  your  overall  experiences  and  judgments 
about  the  simulations  that  you  just  completed. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and 
to  encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Controller 
A,  B,  C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are 
encouraged  to  write  any  additional  comments  you  feel  are  important. 
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Air-Ground  Integration  Experiment  (AGIE) 
Controller  Exit  Form 


Controller  ID#:  Position:  R-Side  or  D-Side  {circle  one) 

Scenario: _ 

Rui^: _  Sector: _ 

Date: _ ! _ / _ 

Where  applicable,  please  circle  the  most  accurate  response. 

1 .  What  additional  procedures  would  you  suggest  to  facilitate  the  implementation  of  shared- 
separation  operations? 


2.  For  each  control  condition,  what  additional  information  and  decision  support  tools  would 
help  you  under  the  following  operational  conditions  please  rate  the  effectiveness  of  the 
support  tool  (i.e.,  URET)  for  conducting  air  traffic  operations. 


Current 

operations  (CO) 
-  controller  has 
full  authority 

Current 

operations  with 
CDTI 

(COrCDTI)  - 
controller  has 
full  authority 

Shared- 

separation 

authority 

(SS:L1) 
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Flight  deck  has 
full  separation 
authority 
(SS:L2) 


3.  What  additional  information  (or  tools)  would  be  useful  for  shared-separation  operations? 


4.  Rate  the  realism  of  the  simulated  flight  crew  responses  compared  to  your  field  experience. 


1 

2 

3 

4 

5 

Very 

Unrealistic 

Moderate 

Very  Realistic 

Comments. 

5.  Rate  the  overall  realism  of  the  simulation  compared  to  your  field  experience. 


1 

2 

3 

4 

5 

Very 

Unrealistic 

Moderate 

Veiy  Realistic 

Comments. 

6.  Rate  the  adequacy  of  the  simulation  training. 

1  2  3  4  5 

Inadequate  Moderate  Adequate 

Comments. 


7.  Were  you  anytime  confused  about  who  has  the  separation  responsibility  and  what  your  role 
is?  YES  NO  If  Yes,  please  explain 


8.  Did  you  find  scenarios  similar  (except  for  training)?  YES 


NO  If  YES,  describe  how. 


9.  Were  you  able  to  keep  up  with  providing  three  data  points  A,  B,  C  under  SS:L1  and  SS:L2 
conditions?  YES  NO,  If  NO,  describe  below  under  which  locus  it  was  difficult? 


10.  What  can  be  done  to  improve  simulation  fidelity?  If  we  were  to  conduct  future  shared- 
separation  related  research,  what  improvements  in  scenario,  traffic,  phraseology,  and 
simulation  would  you  suggest? 
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Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (CO) 

This  form  is  to  be  completed  by  air  traffic  controllers  participating  in  AGIE  and 
requests  information  regarding  your  overall  experiences  and  judgments  about  the 
run. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and 
to  encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Controller 
A,  B,  C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are 
encouraged  to  write  any  additional  comments  you  feel  are  important. 
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Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (CO) 

Controller  ID#: _ 

Position:  R-Side  or  D-Side  {circle  one) 

Scenario: 

Run#: 

Sector: 

Date:  / 

/ 

Where  applicable,  please  circle  the  most  accurate  response. 


1 .  Rate  your  physical  workload  level  (e.g.,  data  entry,  coordination,  communications,  etc.) 


during  this  run. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

2.  Rate  your  mental  workload  level  (e.g.,  planning,  predicting,  monitoring,  etc.)  during  this 


run. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

3.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


4.  Identify  events  that  significantly  affected  your  overall  workload  during  the  run. 
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5.  Rate  the  overall  workload  associated  with  maintaining  aircraft  separation,  (e.g.. 


monitoring  and  planning). 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

6.  Rate  the  overall  workload  associated  with  maintaining  ground-to-ground  (i.e.,  land  line) 
coordination  /  communications. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


7.  Rate  the  overall  workload  associated  with  maintaining  coordination  /  commimications 


between  R-side  and  D-side. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

8.  Rate  the  overall  workload  associated  with  maintaining  ground-to-air  communications. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


9.  Rate  the  overall  workload  associated  with  coordinating  URET  alerts  to  adjacent  sectors. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


c-\y 


10.  Rate  the  amount  of  time  available  to  assure  safe  separation  during  peak  traffic  periods. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


1 1 .  Rate  the  amount  of  time  available  to  complete  all  required  coordination  during  peak 


traffic  periods. 

1 

2 

3 

4 

5 

Too  Little 
Comments. 

Adequate 

Too  Much 

12.  During  the  peak  traffic  period,  I  felt  rushed. 

12  3  4 

Not  at  All  About  Half  of 

the  Time 

Comments. 


5 

All  of  the 
Time 


13.  During  the  peak  traffic  period,  I  felt  bored. 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 


14.  Compared  to  current  operations  the  procedures  used  in  this  session  resulted  in  a  level  of 


safety  that  was: 

1  2 

3 

4 

5 

Compromised 

If  enhanced,  why? 

Unchanged 

Enhanced 

If  compromised,  why? 
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The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  controller’s 
“picture”  and  involves  processing  the  relevant  air  traffic  information  to  develop  a  thorough 
understanding  of  the  current  situation  that  facilitates  appropriate  air  traffic  actions  in  a  timely 
manner.  _ 


15.  Rate  your  level  of  overall  situation  awareness  during  this  run. 


1 

2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

16.  Rate  the  amount  of  the  information  available  to  identify  and  resolve  conflicts. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


•  What  additional  information  is  needed,  if  any? 


•  What  information  was  not  useful  or  added  clutter,  if  any? 


1 7.  How  timely  were  the  conflict  probe  alerts? 

1  2  3  4  5 

Too  Early  Adequate  Too  Late 

Comments. 


18.  Identify  factors  that  interfered  with  maintaining  separation. 


19.  Identify  any  changes  in  your  control  strategies  for  this  run. 
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20.  In  your  opinion,  how  much  vertical  and  horizontal  separation  represents  effective  safety 
margin? 

Vertical  separation  (below  FL290): _ (ft) 

Vertical  separation  (above  FL290): _ (ft) 

Horizontal  separation: _ (nm) 

21 .  Please  provide  any  additional  comments  or  concerns  you  may  have  about  this  run. 


Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (COrCDTI) 

This  form  is  to  be  completed  by  air  traffic  controllers  participating  in  AGIE  and 
requests  information  regarding  your  overall  experiences  and  judgments  about  the 
run. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and 
to  encourage  unbiased  reporting.  Findings  will  be  reported  geierically  (Controller 
A,  B,  C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are 
encouraged  to  write  any  additional  comments  you  feel  are  important. 
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Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (CO:CDTI) 

Controller  ID#: _ 

Position:  R*Side  or  D-Side  {circle  one) 

Scenario: 

Run#: 

Sector: 

Date:  / 

/ 

Where  applicable,  please  circle  the  most  accurate  response. 


1 .  Rate  your  physical  workload  level  (e.g.,  data  entry,  coordination,  communications,  etc.) 


during  this  run. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

2.  Rate  your  mental  workload  level  (e.g.,  planning,  predicting,  monitoring,  etc.)  during  this 


run. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

3.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


4.  Identify  events  that  significantly  affected  your  overall  workload  during  the  nm. 
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5.  Rate  the  overall  workload  associated  with  maintaining  aircraft  separation,  (e.g.. 


monitoring  and  planning). 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

6.  Rate  the  overall  workload  associated  with  maintaining  ground-to-ground  (i.e.,  land  line) 


coordination/communications. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

7.  Rate  the  overall  workload  associated  with  maintaining  coordination/communications 


between  R-side  and  D-side. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

8.  Rate  the  overall  workload  associated  with  maintaining  ground-to-air  communications. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


9  Rate  the  overall  workload  associated  with  coordinating  URET  alerts  to  adjacent  sectors. 
’  1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 
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10.  Rate  the  amount  of  time  available  to  assure  safe  separation  during  peak  traffic  periods. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


1 1 .  Rate  the  amount  of  time  available  to  complete  all  required  coordination  during  peak 


traffic  periods. 

1 

2 

3 

4 

5 

Too  Little 
Comments. 

Adequate 

Too  Much 

12.  During  the  peak  traffic  period,  I  felt  rushed: 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 


13.  During  the  peak  traffic  period,  I  felt  bored: 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 


14.  Compared  to  current  operations  the  procedures  used  in  this  session  resulted  in  a  level  of 


safety  that  was: 

1  2 

3 

4 

5 

Compromised 

Unchanged 

Enhanced 

If  enhanced,  why? 

If  compromised,  why? 


C-20 


The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  controller’s 
“picture”  and  involves  processing  the  relevant  air  traffic  information  to  develop  a  thorough 
understanding  of  the  current  situation  that  facilitates  appropriate  air  traffic  actions  in  a  timely 
manner.  _ 

15.  Rate  your  level  of  overall  situation  awareness  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Conmients. 


16.  Rate  the  amount  of  the  information  available  to  identify  and  resolve  conflicts. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


•  What  additional  information  is  needed,  if  any? 


•  What  information  was  not  useful  or  added  clutter,  if  any? 


17.  How  timely  were  the  conflict  probe  alerts? 

1  2  3  4  5 

Too  Early  Adequate  Too  Late 

Comments. 


18.  Identify  factors  that  interfered  with  maintaining  separation. 


19.  Identify  any  changes  in  your  control  strategies  for  this  run. 
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20.  In  your  opinion,  how  much  vertical  and  horizontal  separation  represents  effective  safety 
margin? 

Vertical  separation  (below  FL290): _ (ft.) 

Vertical  separation  (above  FL290): _ (ft.) 

Horizontal  separation: _ (nm) 

21 .  Please  provide  any  additional  comments  or  concerns  you  may  have  about  this  run. 


Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (SS:L1) 

This  form  is  to  be  completed  by  air  traffic  controllers  participating  in  AGIE  and 
requests  information  regarding  your  overall  experiences  and  judgments  about  the 
run. 


Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and 
to  encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Controller 
A,  B,C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are 
encouraged  to  write  any  additional  comments  you  feel  are  important. 


Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (SS:L1) 

Controller  ID#:  Position:  R-Side  or  D-Side  {circle  one) 

Scenario: _ 

Run#: _  Sector: _ 

Date: _ t _ ! _ 

Where  applicable,  please  circle  the  most  accurate  response. 

1 .  What  changes  would  you  recommend  for  the  implementation  of  Free  Flight  as  simulated 
in  this  study? 

Automation. 


Procedures. 
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2.-  Rate  your  physical  workload  level  (e.g.,  data  entry,  coordination,  communications,  etc.) 


during  this  run. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

3.  Rate  your  mental  workload  level  (e.g.,  planning,  predicting,  monitoring,  etc.)  during  this 


run. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

4.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


5.  Identify  events  that  significantly  affected  your  overall  workload  during  the  run. 


6.  Rate  the  overall  workload  associated  with  maintaining  aircraft  separation,  (e.g.. 


monitoring  and  planning). 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 
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7.  Rate  the  overall  workload  associated  with  maintaining  ground-to-ground  (i.e.,  land  line) 


coordination  /  communications. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

8.  Rate  the  overall  workload  associated  with  maintaining  coordination/communications 


between  R-side  and  D-side. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

9.  Rate  the  overall  workload  associated  with  maintaining  ground-to-air  communications. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


10.  Rate  the  overall  workload  associated  with  coordinating  URET  alerts  to  adjacent  sectors. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


1 1 .  Rate  the  amount  of  time  available  to  assure  safe  separation  during  peak  traffic  periods. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 
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12.  Rate  the  amount  of  time  available  to  complete  all  required  coordination  during  peak 


traffic  periods. 

1 

2 

3 

4 

5 

Too  Little 
Comments. 

Adequate 

Too  Much 

13.  During  the  peak  traffic  period,  I  felt  rushed. 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 


14.  During  the  peak  traffic  period,  I  felt  bored. 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Corrunents. 


15.  Compared  to  current  operations  the  procedures  used  in  this  session  resulted  in  a  level  of 


safety  that  was: 

1  2 

3 

4 

5 

Compromised 

If  enhanced,  why?. 

Unchanged 

Enhanced 

If  compromised,  why? 
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The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  controller’s 
“picture”  and  involves  processing  the  relevant  air  traffic  information  to  develop  a  thorough 
understanding  of  the  current  situation  that  facilitates  appropriate  air  traffic  actions  in  a  timely 
manner. 

16.  Rate  your  level  of  overall  situation  awareness  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


17.  Rate  the  amount  of  the  information  available  to  identify  and  resolve  conflicts. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


•  What  information  was  not  useful  or  added  clutter,  if  any? 


18.  How  timely  were  the  conflict  probe  alerts? 

1  2  3  4  5 

Too  Early  Adequate  Too  Late 

Comments. 


19.  Identify  factors  that  interfered  with  maintaining  separation. 


20.  Identify  any  changes  in  your  control  strategies  for  this  run. 
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21.  In  your  opinion,  how  much  vertical  and  horizontal  separation  represents  effective  safety 
margin? 

Vertical  separation  (below  FL290): _ (ft) 

Vertical  separation  (above  FL290): _ (ft) 

Horizontal  separation: _ (nm) 


22.  Rate  how  often  you  monitored  air-to-air  communications. 
12  3  4 

Never  Sometimes 

Comments. 


5 

Always 


23.  Rate  the  usefulness  of  monitoring  air-to-air  communications. 

12  3  4 

Not  Useful  Moderately 

Useful 

Comments. 


Very  Useful 


24.  Rate  the  helpfulness  of  the  shared-separation  operations  concept  for  performing  your  job. 

1  2  3  4  5 

Not  Helpful  Moderately  Very  Helpful 

Helpful 

Conunents. 


25.  If  you  cancelled  Free  Flight,  identify  the  reasons. 


If  you  delayed  in  resuming  Free  Flight,  identify  the  reasons, 


26.  Please  provide  any  additional  comments  or  concerns  you  may  have  about  this  run. 


Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (SS:L2) 

This  form  is  to  be  completed  by  air  traffic  controllers  participating  in  AGIE  and 
requests  information  r^arding  your  overall  experiences  and  judgments  about  the 
run. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and 
to  encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Controller 
A,  B,  C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are 
encouraged  to  write  any  additional  comments  you  feel  are  important. 
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Air-Ground  Integration  Experiment  (AGIE) 

Controller  Post-Run  Form  (SS:L2) 

Controller  DD#: _ 

Position:  R-Side  or  D-Side  (circle  one) 

Scenario: 

Rui^: 

Sector: 

Date:  / 

/ 

Where  applicable,  please  circle  the  most  accurate  response. 


1 .  What  changes  would  you  recommend  for  the  implementation  of  Free  Flight  as  simulated 
in  this  study? 

Automation. 


Procedures. 


2.  Rate  your  physical  workload  level  (e.g.,  data  entry,  coordination, 


during  this  run. 

1 

2 

3 

Very  Low 

Moderate 

Comments. 

communications,  etc.) 
5 

Very  High 


3.  Rate  your  mental  workload  level  (e.g.,  planning,  predicting,  monitoring,  etc.)  during  this 


run. 

1 

2 

3 

4 

5 

Very  Low 

Moderate 

Veiy  High 

Comments. 

4.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


5.  Identify  events  that  significantly  affected  your  overall  workload  during  the  run. 


6.  Rate  the  overall  worittoarf  associated  with  maintaining  aircraft  separation,  (e.g.. 


monitoring  and  planning). 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 
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7.  Rate  the  overall  workload  associated  with  maintaining  ground-to-ground  (i.e.,  land  line) 


coordination/communications. 

1  2 

3 

4 

5 

Very  Low 

Comments. 

Moderate 

Very  High 

8.  Rate  the  overall  workload  associated  with  maintaining  coordination/communications 


between  R-side  and  D-side. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

9.  Rate  the  overall  workload  associated  with  maintaining  ground-to-air  communications. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


10.  Rate  the  overall  workload  associated  with  coordinating  URET  alerts  to  adjacent  sectors. 


1 

2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

1 1 .  Rate  the  amount  of  time  available  to  assure  safe  separation  during  peak  traffic  periods. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 
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12.  Rate  the  amount  of  time  available  to  complete  all  required  coordination  during  peak 
traffic  periods. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


13.  During  the  peak  traffic  period,  1  felt  rushed. 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 


14.  During  the  peak  traffic  period,  I  felt  bored. 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 


15.  Compared  to  current  operations  the  procedures  used  in  this  session  resulted  in  a  level  of 


safety  that  was: 

1  2 

3 

4 

5 

Compromised 

Unchanged 

Enhanced 

If  enhanced,  why? 

If  compromised,  why? 


The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  controller’s 
“picture”  and  involves  processing  the  relevant  air  traffic  information  to  develop  a  thorough 
understanding  of  the  current  situation  that  facilitates  appropriate  air  traffic  actions  in  a  timely 
manner. 


16.  Rate  your  level  of  overall  situation  awareness  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


17.  Rate  the  amount  of  the  information  available  to  identify  and  resolve  conflicts. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


•  What  additional  information  is  needed,  if  any? 


•  What  information  was  not  useful  or  added  clutter,  if  any? 


18.  How  timely  were  the  conflict  probe  alerts? 

1  2  3  4  5 

Too  Early  Adequate  Too  Late 

Comments. 


19.  Identify  factors  that  interfered  with  maintaining  separation. 


20.  Identify  any  changes  in  your  control  strategies  for  this  run. 
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21.  In  your  opinion,  how  much  vertical  and  horizontal  separation  represents  effective  safety 
margin? 

Vertical  separation  (below  FL290): _ (ft) 

Vertical  separation  (above  FL290): _ (ft) 

Horizontal  separation: _ (nm) 

22.  Rate  how  often  you  monitored  air-to-air  communications. 

1  2  3  4  5 

Never  Sometimes  Always 

Comments. 


23.  Rate  the  usefulness  of  monitoring  air-to-air  communications. 

1  2  3  4  5 

Not  Useful  Moderately  Very  Useful 

Useful 

Comments. 


24.  Rate  the  helpfulness  of  the  shared-separation  operations  concept  for  performing  your  job. 

1  2  3  4  5 

Not  Helpful  Moderately  Very  Helpful 


Helpful 


Comments. 


Air-Ground  Integration  Experiment  (AGIE) 
Observer  During-the-Run  Form 


This  form  is  to  be  completed  by  observers  participating  in  AGIE  and  requests 
information  regarding  your  overall  observations  and  judgments  during  the  run. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and 
to  encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Observer 
A,  B,  C,  etc.). 

You  are  encouraged  to  write  any  additional  comments  that  you  feel  are  important. 

Please  review  the  items  contained  in  the  questionnaire  before  the  simulation  stots. 
Listen  and  closely  observe  the  actions  of  the  controller  team  operating  in  your 
sector(s).  Based  on  what  you  hear  and  see,  apply  your  expertise  to  provide 
information  and  comments  that  you  feel  may  be  valuable  to  this  study. 
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AGIE  -Observer  During-the-Run  Form 


Observer  ID#: 
Scenario: _ 


Controller  IDs,  R-Side: _ 

Run  #: _  Sector: 


D-Side: 


Record  of  Conflict  Detection  and  Resolution  Points 


URET 

Point  A 

Point  B 

Point  C 

Aircraft  1  ID 

Aircraft  2  ID 

Comments 

U 

A 

B 

c 

Sim  Time 

U 

A 

B 

c 

Sim  Time 

U 

A 

B 

c 

Sim  Time 

U 

A 

B 

c 

Sim  Time 

u 

A 

B 

c 

Sim  Time 

U 

A 

B 

c 

Sim  Time 

u 

A 

B 

c 

Sim  Time 

U 

A 

B 

c 

Sim  Time 

U 

A 

B 

c 

Sim  Time 

Aircraft  IDs 

Simulation  Time 
of  Free  Flight 
Cancellation 

lit. 

Simulation  Time 
that  Free  Flight 
Resumed 

Free  Flight 
Cancelled  by: 
Pilot  or 
Controller 

Why  was  Free  Flight  Canceiled? 

P  C 

P  C 

P  c 

P  c 

P  c 

P  c 

Record  of  Situation  Awareness,  Separation  Errors  &  Airspace  Violations  Events 
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Please  record  the  time  of  the  following  events  as  and  when  you  notice  (record  the  time  of 

_ the  event,  aircraft  ID,  and  put  a  check  mark  in  the  appropriate  colunm) _ 

Simulation  Aircraft  ID  Separation  Airspace  Missed  Late  Failure  to  Others  (Specify) 

Time  Error  Violations  Handoffs  Recognition  of  Correct 


Conflicts  Readback 
Errors 
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As  and  when  appropriate,  please  record  vour  observations  regarding  the  followinj 

1.  UseofURET 

2.  Coordination  between  R  and  D  side  controllers 

3.  Coordination  with  other  sectors 

4.  Phraseology 

5.  Difficulty  with  simulation  environment,  equipment,  voice  system,  etc. 

6.  Air-to-air  frequency  usage  (for  SS:L1  and  SS:L2  only) 

7.  Other  relevant  information 
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Appendix  D 

Human  Research  Minimal  Risk  Consent 


NASA  Ames  Research  HUMAN  RESEARCH 

_ _ MINIMAL  RISK  CONSENT 

To  the  Test  Subject:  Please  read  this  consent  form  and  the  attached  protocol  and/or  subject 

instructions  carefully.  Make  sure  all  your  questions  have  been  answered  to  your 

satisfaction  before  signing. 

A.  I  agree  to  participate  as  a  subject  in  the 

_ research  experiment  as 

described  in  the  attached  protocol  or  subject  instructions.  I  understand  that  I  am 
employed  by _ 

who  can  be  contacted  at _ . 

B.  1  understand  that  my  participation  could  cause  me  minimal  risk*,  inconvenience,  or 
discomfort.  The  purpose  and  procedures  have  been  explained  to  me  and  I  understand 
the  risks  and  discomforts  as  described  in  the  attached  research  protocol. 

C.  To  my  knowledge,  I  have  no  medical  conditions,  including  pregnancy,  that  will 
prevent  my  participation  in  this  study.  I  understand  that  if  my  medical  status  should 
change  while  a  participant  in  the  research  experiment  that  there  may  be  unforeseeable 
risks  to  me  (or  the  embryo  or  fetus  if  applicable).  1  agree  to  notify  the  Principal 
Investigator  (P.I.)  or  medical  monitor  of  any  known  changes  in  my  condition  for 
safety  purposes. 

D.  My  consent  to  participate  as  a  subject  has  been  freely  given.  I  may  withdraw  my 
consent,  and  thereby  withdraw  from  the  study  at  any  time  without  penalty  or  loss  of 
benefits  to  which  I  am  entitled.  I  understand  that  the  P.I.  may  request  my  withdrawal 
or  the  study  may  be  terminated  for  any  reason.  I  agree  to  follow  procedures  for 
orderly  and  safe  termination. 

E.  I  am  not  releasing  NASA  from  liability  for  any  injury  arising  as  a  result  from  my 
participation  in  this  study. 

F.  I  hereby  agree  that  all  records  collected  by  NASA  in  the  course  of  this  study  are 
available  to  the  research  study  investigators,  support  staff,  and  any  duly  authorized 
research  review  committee.  I  grant  NASA  permission  to  reproduce  and  publish  all 
records,  notes,  or  data  collected  from  my  participation,  provided  there  is  no 
association  of  my  name  with  the  collected  data  and  that  confidentiality  is  maintained, 
unless  specifically  waived  by  me. 

G.  I  have  had  an  opportunity  to  ask  questions  and  have  received  satisfactory  answers  to 
all  my  questions.  I  understand  the  P.I.  for  the  study  is  the  person  responsible  for  this 
activity  and  that  any  pertinent  questions  regarding  the  research  will  be  addressed  to 
him/her  during  the  course  of  the  study.  I  have  read  the  above  agreement,  the  attached 
protocol  and/or  subject  instructions  prior  to  signature,  and  understand  the  contents. 
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*  Minimal  Risk  means  that  the  probability  and  magnitude  of  harm  or  discomfort 

anticipated  in  the  research  are  not  greater,  in  and  of  themselves,  than  those  ordinarily 
encountered  in  daily  life  or  during  the  performance  of  routine  physical  or 
psychological  examinations  or  tests. 

Signature  of  Test  Subject 

Date 

Signature  of  Principal  Investigator 

Date 

Printed/Typed  Name  of  Test 
Subject/Evaluation  Pilot 

Printed/Typed  Name  of  Principal 

Investigator 

Address 

Telephone  Number  of  Principal 

Investigator 

City,  State,  Zip  Code 

Subject  Signature:  Authorization  for 
Videotaping 

Telephone  Number  of  Test  Subject 

Subject  Signature:  Authorization  for 

Release  of  Information  to  Non-NASA 
Source(s) 
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Appendix  E 
Pilot  Daily  Schedule 


Daily  Schedule  for  Pilot  Participants 


Time  (PST) 

Day  2 

Days 

Day  4 

Days 

09:00am 

Pilot 

Review 

:15 

Briefing 

Review 

:30 

:45 

10:00am 

Pilot 

Run  3 

:15 

Training 

:30 

:45 

1 1:00am 

Forms  & 

:I5 

Discussion 

:30 

:45 

MEAL 

12:00pm 

MEAL 

BREAK 

:15 

BREAK 

:30 

:45 

Pilot 

Pilot 

01:00pm 

Travel 

Run  4 

Travel 

:15 

Run  1 

:30 

:45 

02:00pm 

Forms  & 

:15 

Forms  & 

Discussion 

:30 

Discussion 

BREAK 

:45 

BREAK 

03:00pm 

Debriefing 

:15 

:30 

Run  2 

:45 

Buffer 

04:00pm 

Buffer 

:15 

Buffer 

:30 

Forms  & 

Buffer 

:45 

Discussion 

Buffer 
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Appendix  F 
Pilot  Forms 


Air  /  Ground  Integration  Experiment  (AGIE) 

Flight  Crew  Post-Run  Form 

This  form  is  to  be  completed  by  pilots  participating  in  AGIE  and  requests  information 
regarding  your  overall  experiences  and  judgments  about  the  simulation. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and  to 
encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Pilot  A,  B,  C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are  encouraged  to 
write  any  additional  comments  you  feel  are  important. 
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Air  /  Ground  Integration  Experiment  (AGIE) 
Pilot  Post-Run  Form 


Date: _ ! _ / 

Pilot  ID#: _ 

Scenario: 


Position:  Captain  or  First  Officer  {circle  one) 
Run#: 


Where  applicable,  please  circle  the  most  accurate  response. 

1 .  Rate  the  amount  of  uncertainty  (if  any)  about  who  had  authority  (controller,  pilot  or 
both)  for  maintaining  separation  during  the  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

2.  Rate  your  physical  workload  level  (e.g.,  data  entry,  flying  the  aircraft, 
conunimications,  etc.)  during  this  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Conunents. 


3.  Rate  your  mental  workload  level  (e.g.,  plaiming,  predicting,  and  monitoring,  etc.) 


during  this  last  set  of  scenarios. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 
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4.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  last  set 


of  scenarios. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

5.  Identify  events  that  significantly  affected  your  overall  workload  during  this  last  set  of 
scenarios. 


6.  Rate  ihe  overall  workload  associated  with  coordination  and  communication  between 
crew  members. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


7.  Rate  the  overall  worArtocrf  associated  with  maintaining  air-to-ground  communications. 


1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 
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8.  Rate  the  amount  of  time  available  for  air  to  ground  communications. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


9.  Rate  the  amount  of  time  available  for  crew  coordination  and  communication. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


10.  During  the  last  set  of  scenarios,  I  felt  rushed: 

1  2  3  4 

Not  at  All  About  Half  of 

the  Time 

Comments. 


5 

All  of  the 
Time 


1 1 .  During  the  last  set  of  scenarios,  1  felt  bored: 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 
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The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  pilot’s 
“staying  ahead  of  the  aircraft”  where  the  pilot  has  a  thorough  understanding  of  the  current 
situation  and  can  take  appropriate  action  as  necessary. _ 

12.  Rate  your  level  of  overall  situation  awareness  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 


Comments. 


13.  Please  provide  any  additional  comments  or  concerns  you  may  have  about  this 
simulation. 


Captain  /  First  Officer 


ID#: _  Run# 

Air  /  Ground  Integration  Experiment  (AGIE) 

Flight  Crew  Post-Run  Form 

This  form  is  to  be  completed  by  pilots  participating  in  AGIE  and  requests  information 
regarding  your  overall  experiences  and  judgments  about  the  simulation. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and  to 
encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Pilot  A,  B,  C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are  encouraged  to 
write  any  additional  comments  you  feel  are  important. 
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Date: 


Air  /  Ground  Integration  Experiment  (AGIE) 
Pilot  Post-Run  Form 

/  / 


Pilot  ID#: 
Scenario: 


Position:  Captain  or  First  Officer  {circle  one) 
Run#: 


Where  applicable,  please  circle  the  most  accurate  response. 

1 .  Rate  the  amount  of  uncertainty  (if  any)  about  who  had  authority  (controller,  pilot  or 
both)  for  maintaining  separation  during  the  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

2.  Rate  your  physical  workload  level  (e.g.,  data  entry,  flying  the  aircraft, 
communications,  etc.)  during  this  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


3.  Rate  your  mental  workload  level  (e.g.,  planning,  predicting,  and  monitoring,  etc.) 
during  this  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 
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4.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  last  set 


of  scenarios. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

5.  Identify  events  that  significantly  affected  your  overall  workload  during  this  last  set  of 
scenarios. 


6.  Rate  the  overall  workload  associated  with  coordination  and  communication  between 


crew  members 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

7.  Rate  the  overall  workload  associated  with  maintaining  air-to-ground  communications. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 
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8.  Rate  the  amount  of  time  available  for  air  to  ground  communications. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


9.  Rate  the  amount  of  time  available  for  crew  coordination  and  communication. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


10.  How  timely  was  the  conflict  alert? 

1  2  3  4  5 

Too  Early  Adequate  Too  Late 

Comments. 


1 1 .  How  easy  was  it  to  detect  a  potential  conflict  prior  to  a  system  alert  or  a  controller 


traffic  advisory? 

1 

2 

3 

4 

5 

Not  Easy 
Comments. 

Moderately  Easy 

Very  Easy 

4 


12.  During  the  last  set  of  scenarios,  1  felt  rushed: 


1 

Not  at  All 


About  Half  of 
the  Time 


Conunents. 


5 

All  of  the 
Time 


13.  During  the  last  set  of  scenarios,  1  felt  bored: 

1  2  3  4  5 

Not  at  All  About  Half  of  All  of  the 

the  Time  Time 

Comments. 


14.  Compared  to  ciurent  operations  the  procedures  used  in  this  set  of  scenarios  resulted  in 


a  level  of  safety  that  was: 

1  2 

3 

4 

5 

Compromised 

If  enhanced,  why? 

Unchanged 

Enhanced 

If  compromised,  why? 


F-10 


The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  pilot’s 
“staying  ahead  of  the  aircraft”  where  the  pilot  has  a  thorough  understanding  of  the  current 
situation  and  can  take  appropriate  action  as  necessary. 

15.  Rate  your  level  of  overall  situation  awareness  during  this  run. 

1  2  3  4 

Very  Low  Moderate 

Comments. 


16.  Please  estimate  the  percentage  of  time  you  spent  monitoring  the  CDTI:  _ % 

Did  monitoring  the  CDTI  interfere  with  your  other  flight  duties?  Yes _ No _ 


17.  Please  provide  any  additional  comments  or  concerns  you  may  have  about  this 
simulation. 


5 

Very  High 


Captain  /  First  Officer 


ID#: _  Run# 

Air  /  Ground  Integration  Experiment  (AGIE) 

Flight  Crew  Post-Run  Form 

This  form  is  to  be  completed  by  pilots  participating  in  AGIE  and  requests  information 
regarding  your  overall  experiences  and  judgments  about  the  simulation. 

Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and  to 
encourage  unbiased  reporting.  Findings  will  be  reported  generically  (Pilot  A,  B,  C,  etc.). 

When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are  encouraged  to 
write  any  additional  comments  you  feel  are  important. 
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Date: 


Air  /  Ground  Integration  Experiment  (AGIE) 
Pilot  Post-Run  Form 

/  / 


Pilot  ID#: 
Scenario: 


Position:  Captain  or  First  Officer  {circle  one) 
Run#: 


Where  applicable,  please  circle  the  most  accurate  response. 

1 .  Rate  the  amount  of  uncertainty  (if  any)  about  who  had  authority  (controller,  pilot  or 
both)  for  maintaining  separation  during  the  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

2.  Rate  your  physical  workload  level  (e.g.,  data  entry,  flying  the  aircraft, 
communications,  etc.)  during  this  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


3.  Rate  your  mental  workload  level  (e.g.,  planning,  predicting,  and  monitoring,  etc.) 
during  this  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


4.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  last  set 


of  scenarios. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

5.  Identify  events  that  significantly  affected  your  overall  workload  during  this  last  set  of 
scenarios. 


6.  Rate  the  overall  workload  associated  with  maintaining  aircraft  separation,  (e.g., 


monitoring  and  planning). 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

7.  Rate  the  overall  workload  associated  with  maintaining  air-to-air  communications. 

1  2  3  4  5 

Very  Low  Moderate  Veiy  High 

Comments. 
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8.  Rate  the  overall  workload  associated  with  coordination  and  communication  between 


crew  members. 

1 

2 

3 

4 

5 

Very  Low 
Comments. 

Moderate 

Very  High 

9.  Rate  the  overall  workload  associated  with  maintaining  air-to-ground  communications. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


10.  Rate  the  amount  of  time  available  to  detect,  monitor  and  resolve  conflicts  and 
maintain  self-separation  with  other  aircraft. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


1 1 .  Rate  the  amoxmt  of  time  available  for  air  to  air  commimications. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 
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12.  Rate  the  amount  of  time  available  for  air  to  ground  communications. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


13.  Rate  the  amount  of  time  available  for  crew  coordination  and  communication. 
12  3  4 

Too  Little  Adequate 

Comments. 


5 

Too  Much 


14.  How  timely  was  the  conflict  alert? 
1  2 
Too  Early 
Comments. 


3  4 

Adequate 


5 

Too  Late 


15.  How  easy  was  it  to  detect  a  potential  conflict  prior  to  a  system  alert  or  a  controller 
traffic  advisory? 

1  2  3  4  5 

Not  Easy  Moderately  Easy  Very  Easy 

Comments. 
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16.  Please  rate  the  effectiveness  of  the  alerting  logic  and  display  for  conducting  free 
flight  self-separation  operations. 

1  2  3  4  5 

Not  Moderately  Very 

Effective  Effective  Effective 

Comments. 


17.  Identify  factors  that  interfered  with  maintaining  separation  during  this  last  set  of 


scenanos. 


18.  During  the  last  set  of  scenarios,  I  felt  rushed: 

1  2  3 

Not  at  All  About  Half  of 

the  Time 

Comments. 


All  of  the 
Time 


19.  During  the  last  set  of  scenarios,  I  felt  bored: 

1  2  3 

Not  at  All  About  Half  of 

the  Time 


All  of  the 
Time 


Comments 


20.  Compared  to  current  operations  the  procedures  used  in  this  last  set  of  scenarios 


resulted  in  a  level  of  safety  that  was; 


1  2 

3 

4 

5 

Compromised 

If  enhanced,  why?. 

Unchanged 

Enhanced 

If  compromised,  why? 


The  term  overall  situation  awareness  refers  to  what  is  cotiunonly  known  as  the  pilot’s 
“staying  ahead  of  the  aircraft”  where  the  pilot  has  a  thorough  understanding  of  the  current 
situation  and  can  take  appropriate  action  as  necessary.  _ _ _ _ _ 


21  Rate  your  level  of  overall  situation  awareness  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


22.  Rate  how  comfortable  you  were  with  sharing  separation  responsibility  with  the 


controller. 

1 

2 

3 

4 

5 

Not 

Comfortable 

Comments. 

Moderately 

Comfortable 

Very 

Comfortable 

r 
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23.  Rate  how  often  you  monitored  other  air-air  communications. 
I  2  3 

Never  Sometimes 

Comments. 


5 

Always 


24.  Rate  the  usefulness  of  monitoring  other  air-air  communications. 


1  2 
Not  Useful 


3  4 

Moderately 
Useful 


Comments. 


5 

Very  Useful 


25.  Rate  the  impact  of  the  shared-separation  operations  concept  for  performing  your  job. 

1  2  3  4  5 

Detrimental  No  Impact  Helpful 

Comments. 


26.  Please  estimate  the  percentage  of  time  you  spent  monitoring  the  CDTI:  _ % 

Did  monitoring  the  CDTI  interfere  with  your  other  flight  duties?  Yes _ No _ 
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27.  If  you  cancelled  Free  Flight,  identify  the  reasons. 


28.  Rate  the  usefulness  of  traffic  advisories  that  were  issued  by  the  controller. 


1 

2 

3 

4 

5 

Not  Useful 

Moderately 

Very  Useful 

Useful 

29.  Please  provide  any  additional  comments  or  concerns  you  may  have  about  this 
simulation. 
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Captain  /  First  Officer 


ID#:  Run# 


Air  /  Ground  Integration  Experiment  (AGIE) 
Flight  Crew  Post-Run  Form 


This  form  is  to  be  completed  by  pilots  participating  in  AGIE  and  requests  information 
regarding  your  overall  experiences  and  judgments  about  the  simulation. 


Your  name  will  not  be  listed  or  appear  in  any  reports  to  ensure  your  anonymity  and  to 
encourage  unbiased  reporting.  Findings  will  be  reported  genetically  (Pilot  A,  B,  C,  etc.). 


When  making  your  ratings,  please  consider  all  levels  of  the  scale.  You  are  encouraged  to 
write  any  additional  comments  you  feel  are  important. 


F-21 


Date: 


Air  /  Ground  Integration  Experiment  (AGIE) 
Pilot  Post-Run  Form 


/ 


Pilot  ID#; 
Scenario: 


Position;  Captain  or  First  Officer  {circle  one) 
Run#: 


Where  applicable,  please  circle  the  most  accurate  response. 

1 .  Rate  the  amount  of  uncertainty  (if  any)  about  who  had  authority  (controller,  pilot  or 
both)  for  maintaining  separation  during  the  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

2.  Rate  your  physical  workload  level  (e.g.,  data  entry,  flying  the  aircraft, 
communications,  etc.)  during  this  last  set  of  scenarios. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


3.  Rate  your  mental  workload  level  (e.g.,  planning,  predicting,  and  monitoring,  etc.) 


during  this  last  set  of  scenarios. 

1  2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 
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4.  Rate  your  overall  workload  level  (physical  and  mental  combined)  during  this  last  set 
of  scenarios. 


1  2 
Very  Low 
Comments. 


3 

Moderate 


4  5 

Very  High 


5.  Identify  events  that  significantly  affected  your  overall  workload  during  this  last  set  of 
scenarios. 


6.  Rate  the  overall  workload  associated  with  maintaining  aircraft  separation,  (e.g., 


monitoring  and  planning). 

1  2 

3 

4 

5 

Very  Low 

Comments. 

Moderate 

Very  High 

7.  Rate  the  overall  workload  associated  with  maintaining  air-to-air  communications. 


1 

2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 
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8.  Rate  the  overall  workload  associated  with  coordination  and  communication  between 
crew  members. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


9.  Rate  the  overall  workload  associated  with  maintaining  air-to-ground  communications. 


1 

2 

3 

4 

5 

Very  Low 

Moderate 

Very  High 

Comments. 

10.  Rate  the  amount  of  time  available  to  detect,  monitor  and  resolve  conflicts  and 
maintain  self-separation  with  other  aircraft. 

1  2  3  4 

Too  Little  Adequate 

Comments. 


5 

Too  Much 


1 1 .  Rate  the  amount  of  time  available  for  air  to  air  communications. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 
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12.  Rate  the  amount  of  time  available  for  air  to  ground  communications. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


13.  Rate  the  amount  of  time  available  for  crew  coordination  and  communication. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 

Comments. 


14.  How  timely  was  the  conflict  alert? 

1  2 

3 

4 

5 

Too  Early 

Adequate 

Too  Late 

Comments. 

15.  How  easy  was  it  to  detect  a  potential  conflict  prior  to  a  system  alert  or  a  controller 


traffic  advisory? 

1 

2 

3 

4 

5 

Not  Easy 
Comments. 

Moderately  Easy 

Very  Easy 
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16.  Please  rate  the  effectiveness  of  the  alerting  logic  and  display  for  conducting  free 
flight  self-separation  operations. 

1  2  3  4  5 

Not  Moderately  Very 

Effective  Effective  Effective 

Comments. 


17.  Identify  factors  that  interfered  with  maintaining  separation  during  this  last  set  of 


scenanos. 


18.  During  the  last  set  of  scenarios,  I  felt  rushed: 

1  2  3 

Not  at  All  About  Half  of 

the  Time 

Comments. 


All  of  the 
Time 


19.  During  the  last  set  of  scenarios,  I  felt  bored: 

1  2  3 

Not  at  All  About  Half  of 

the  Time 


All  of  the 
Time 


Conunents. 


20.  Compared  to  current  operations  the  procedures  used  in  this  set  of  scenarios  resulted  in 


a  level  of  safety  that  was: 

1  2 

3 

4 

5 

Compromised 

If  enhanced,  why? 

Unchanged 

Enhanced 

If  compromised,  why? 


The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  pilot’s 
“staying  ahead  of  the  aircraft”  where  the  pilot  has  a  thorough  understanding  of  the  current 
situation  and  can  take  appropriate  action  as  necessary. 


21.  Rate  your  level  of  overall  situation  awareness  during  this  run. 

1  2  3  4  5 

Very  Low  Moderate  Very  High 

Comments. 


22.  Rate  how  comfortable  you  were  with  sharing  separation  responsibility  with  the 


controller. 

1 

2 

3 

4 

5 

Not 

Comfortable 

Comments. 

Moderately 

Comfortable 

Very 

Comfortable 
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23.  Rate  how  often  you  monitored  other  air-air  communications. 
1  2  3 

Never  Sometimes 

Comments. 


5 

Always 


24.  Rate  the  usefulness  of  monitoring  other  air-air  communications. 

1  2  3  4  5 

Not  Useful  Moderately  Very  Useful 

Useful 

Comments. 


25.  Rate  the  impact  of  the  shared-separation  operations  concept  for  performing  your  job. 

1  2  3  4  5 

Detrimental  No  Impact  Helpful 

Comments. 


26.  Please  estimate  the  percentage  of  time  you  spent  monitoring  the  CDTl:  _ % 

Did  monitoring  the  CDTI  interfere  with  your  other  flight  duties?  Yes _ No _ 
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27.  If  you  cancelled  Free  Flight,  identify  the  reasons. 


29.  Please  provide  any  additional  comments  or  concerns  you  may  have  about  this 
simulation. 


Air  /  Ground  Integration  Experiment  (AGIE) 

Pilot  Exit  Form 


Date: _ ! _ ! _ 

Captain  or  First  Officer  {circle  one) 


Total  Flying  Hours: _  Total  Hours  on  B747-400: 

Where  applicable,  please  circle  the  most  accurate  response. 

1.  Rate  the  adequacy  of  the  simulation  briefing. 

12  3  4 

Not  Effective  Adequate 


5 

Very 

Effective 


Comments. 


2.  Rate  the  adequacy  of  the  simulation  training. 

1  2  3 

Not  Effective  Adequate 

Comments. 


Very 

Effective 


3.  Please  rate  the  effectiveness  of  the  CDTI  and  alerting  system  for  safe  flight  operations. 

1  2  3  4  5 

Not  Effective  Moderately  Very 

Effective  Effective 

Comments. 
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4.  Rate  the  amount  of  the  information  on  the  CDTI  available  to  identify  and  resolve 
conflicts. 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 


Comments. 


5.  How  much  time  did  the  alerting  system  provide  for  strategic  self-separation? 

1  2  3  4  5 

Too  Little  Adequate  Too  Much 


Comments. 


7.  What  information  was  not  useful  or  added  clutter,  if  any? 


8.  What  strategies  did  you  use  to  de-clutter  the  display? 


9.  Identify  your  strategies  for  self-separation. 


10.  During  any  of  the  runs  were  you  ever  unsure  about  which  of  the  authority  conditions 
(e.g.  Shared  authority)  you  were  flying  under?  Please  rate  your  level  of  uncertainty. 

1  2  3  4  5 

Never  Occasionally  Frequently 

Uncertain  Uncertain  Uncertain 

Comments. 
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1 1 .  In  your  opinion  how  much  vertical  and  lateral  separation  represents  an  effective 
safety  margin?  Please  specify  in  terms  of  nautical  miles  and  feet. 

_ vertical  (ft.)  _ horizontal  (nm) 


Comments. 


12.  What  additional  pilot  training  or  skills  (if  any)  do  you  think  might  be  necessary  in  a 
shared  and  flight  deck  authority  enviromnent  compared  with  an  ATC  authority 
enviroiunent? 


13.  What  problems  (if  any)  do  you  anticipate  in  negotiating  separation  maneuvers  with 
other  aircraft  in  conflict  with  you? 


14.  What  procedures  would  you  suggest  to  facilitate  the  implementation  of  shared- 
separation  operations? 
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15.  Rate  the  usefulness  of  the  right-of-way  rules  in  negotiation  and  initiating  contact  with 
a  conflicting  aircraft? 


1 

2 

3 

4 

5 

Not  Useful 

Moderately 

Very  Useful 

Useful 

Comments. 

The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  pilot’s 
“staying  ahead  of  the  aircraft”  where  the  pilot  has  a  thorough  understanding  of  the  current 
situation  and  can  take  appropriate  action  as  necessary. 


16.  Rate  the  usefulness  of  the  air  to  air  frequency  maintaining  situation  awareness  (see 
definition  above)? 


1 

2 

3 

4 

5 

Not  Useful 

Moderately 

Very  Useful 

Useful 

Comments. 

17.  How  distracting  was  the  conflict  detection  button  (EVENT  RCD)  to  your  overall 
task? 

1  2  3  4  5 

Not  At  All  Somewhat  Very 

Distracting  Distracting  Distracting 

Comments. 
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18.  How  representative  do  you  think  your  use  of  the  conflict  detection  button  (EVENT 
RCD)  reflects  the  true  timing  of  when  you  detected  a  conflict? 

1  2  3  4  5 

Not  Somewhat  Very 

Representative  Representative  Representative 

Comments. 


19.  What  can  be  done  to  improve  simulation  fidelity?  If  we  were  to  conduct  future 
shared-separation  related  research,  what  improvements  in  scenario,  traffic,  phraseology, 
and  simulation  would  you  suggest? 


20.  What  additional  information  and  decision  support  tools  do  you  recommend  under  the 
following  flight  authority  conditions: 

Current 

Operations _ 

Controller 

Authority _ 

Shared 

Authority _ 

Flight  Deck 

Authority _ 
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21 .  Did  any  of  the  conflict  scenarios  appear  similar?  Yes 
If  so,  in  regards  to  which  of  the  following  parameters: 

_ conflict  angle 

_ conflict  timing 

_ ownship  flight  plan 

_ intruders  flight  plan 

_ general  location  of  intruder 

_ traffic  pattern 

other _ 


No 
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22.  Please  contrast  these  authority  conditions  in  terms  of  which  was  better  for  flight  safety.  Please  mark  one  box  in  each  line. 


Flight  Deck  Authority  (SS:L2h  Start  in  free  flight  but  can  maneuver  without  informing  the  controller 


23.  Please  contrast  these  authority  conditions  in  terms  of  which  was  better  for  flight  efficiency  (save  fuel,  time  to  destination).  Please 
mark  one  box  in  each  line. 

Absolutely  Much  Better  Slightly  Same  Slightly  Better  Much  Absolutely 

Better  Better  Better  Better  Better  Better 


e 
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24.  Please  contrast  these  authority  conditions  in  terms  of  which  was  better  for  reducing  vour  overall  workload.  Please  mark  one  box 
in  each  line. 
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25.  Please  contrast  these  authority  conditions  in  terms  of  which  was  better  for  maintaining  situation  awareness  (see  definition  below). 
Please  mark  one  box  in  each  line. _ 

The  term  overall  situation  awareness  refers  to  what  is  commonly  known  as  the  pilot’s  “staying  ahead  of  the  aircraft”  where  the  pilot 
has  a  thorough  understanding  of  the  current  situation  and  can  take  appropriate  action  as  necessary. _ 
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Appendix  G 
ANOVA  Results 


Section  G-1.  Controller  Ratines  for  the  Amount  of  Time  Available  to  Assure  Safe 
Aircraft  Separation  and  Complete  Required  Coordination 

The  researchers  performed  two-way  ANOVAs  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for 
the  amount  of  time  available  to  assure  safe  aircraft  separation  and  complete  required 
coordination.  The  ANOVAs  indicated  that  the  main  effect  of  Control  Condition  was 
statistically  significant  for  the  amount  of  time  available  to  assure  safe  aircraft  separation 
[F(3, 27)  =  2.15, p<  .05].  No  other  main  effects  or  interactions  were  significant  for  these 
two  ratings.  The  differences  between  the  four  Control  Condition  means  for  the  amount 
of  time  available  to  assure  safe  aircraft  separation  were  examined  using  Tukey  HSD  post- 
hoc  comparisons.  The  results  indicated  that  significantly  more  time  was  available  for 
CO:CDTI  compared  to  SS:L2,  but  CO:CDTI  was  not  statistically  different  from  CO  or 
SS:L1.  There  were  no  reliable  differences  between  CO,  SS:L1,  and  SS:L2. 

Section  G-2.  Controller  Ratings  for  the  Level  of  Safety  for  Procedures  Compared  to 

Current  Operations 

The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for 
the  level  of  safety  for  procedures  compared  to  current  operations.  The  ANOVA  indicated 
that  the  main  effect  of  Control  Condition  was  statistically  significant  [F(3, 27)  =  27.23, 
p  <  .01].  No  other  main  effects  or  interactions  were  significant  for  these  ratings.  The 
differences  between  the  four  Control  Condition  means  were  examined  using  Tukey  HSD 
post-hoc  comparisons.  The  results  indicated  that  the  level  of  safety  was  significantly 
higher  for  CO  and  COrCDTI  compared  to  SS:L1  and  SS:L2.  There  was  no  reliable 
difference  between  CO  and  CO:CDTI  and  no  difference  between  SS:L1  and  SS:L2. 

Section  G-3A.  Frequency  of  URET  Red  and  Yellow  Alerts 

The  researchers  performed  one-way  ANOVAs  with  only  Control  Condition  (CO, 
CO:CDTI,  SS:L1,  SS:L2)  on  the  frequency  of  URET  red  and  yellow  alerts  (for  each  R- 
Side/D-Side  controller  team).  The  ANOVAs  indicated  that  there  were  no  statistically 
significant  effects  for  the  frequency  of  these  two  alerts. 


% 
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Section  G-3B.  Duration  per  Alert  of  URET  Red  and  Yellow  Alerts 


The  researchers  performed  one-way  ANOVAs  with  only  Control  Condition  (CO, 
CO:CDTI,  SS:L1,  SS:L2)  on  the  duration  per  alert  for  URET  red  and  yellow  alerts  (for 
each  R-Side/D-Side  controller  team).  The  ANOVAs  indicated  that  the  effect  of  Control 
Condition  was  statistically  significant  for  both  the  duration  of  red  alerts  [i*X3, 15)  = 

15.73, p  <  .01],  and  yellow  alerts  [F(3, 15)  =  12.96,p  <  .01].  The  differences  between 
the  four  Control  Condition  means  were  examined  using  Tukey  HSD  post-hoc 
comparisons.  The  results  indicated  that  red  alert  durations  were  longer  in  SS:L2 
compared  to  CO,  CO:CDTI,  and  SS:L1 ,  There  were  no  reliable  differences  between  CO, 
COrCDTI,  and  SS:L1.  Yellow  alert  durations  were  longer  in  SS:L2  compared  to  CO  and 
SS:L1,  but  SS:L2  and  CO:CDTI  were  not  statistically  different.  Also,  yellow  alert 
durations  were  longer  in  COrCDTI  compared  to  SS:L1,  but  COrCDTI  and  CO  were  not 
statistically  different.  Lastly,  there  was  no  reliable  difference  between  CO  and  SS:L1. 

Section  G-4.  Minimum  Separation  Distance  fMSD)  and  Free  Flight  Cancellations 
for  Planned  Conflicts  Involving  WJHTC  Simulation  Pilot 

The  researchers  performed  one-way  ANOVAs  with  only  Control  Condition  (CO, 
CO:CDTI,  SS:L1,  SS:L2)  on  the  MSDs  for  altitude-resolved  and  vector-resolved 
conflicts  (for  each  week  of  two  R-Side/D-Side  controller  teams).  The  ANOVAs 
indicated  that  the  effect  of  Control  Condition  was  statistically  significant  for  the  MSDs  of 
altitude-resolved  conflicts  [F(3, 3)  =  27.40,p  <  .05],  but  not  for  the  MSDs  of  vector- 
resolved  conflicts.  The  differences  between  the  four  Control  Condition  means  for  the 
MSDs  of  altitude-resolved  conflicts  were  examined  using  Tukey  HSD  post-hoc 
comparisons.  The  results  indicated  that  the  MSDs  were  greater  in  CO  and  CO:CDTI 
compared  to  SS:L1  and  SS:L2.  There  was  no  reliable  difference  between  CO  and 
COrCDTI  and  no  difference  between  SS:L1  and  SS:L2. 

Section  G-S.  Controller  Ratines  for  the  Amount  of  Information  Available  to  Resolve 

Conflicts 


The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(CO,  COrCDTI,  SS:L1,  SS;L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for 
the  amount  of  information  available  to  resolve  conflicts.  The  ANOVA  indicated  that 
there  were  no  statistically  significant  main  effects  or  interactions  for  these  ratings. 

Section  G-6.  Controller  Ratines  for  the  URET  Conflict  Alert  Timeliness  of  the 

Conflict  Probes 

The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for 
the  timing  of  the  conflict  probes.  The  ANOVA  indicated  that  there  were  no  statistically 
significant  main  effects  or  interactions  for  these  ratings. 
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Section  G-7.  Controller  Ratines  for  the  Usefulness  of  Monitoring  Air-to-Air 

Communications 


The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(SS:L1,  SS;L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for  the  usefulness  of 
monitoring  air-to-air  communications.  Only  the  ratings  from  controllers  who  monitored 
air-to-air  communications  were  considered  for  this  analysis.  The  ANOVA  indicated  that 
there  were  no  statistically  significant  main  effects  or  interactions  for  these  ratings. 

Section  G-8.  Controller  Ratines  for  the  Helpfulness  of  the  Shared-Seuaration 

Concept 

The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for  the  helpfulness  of 
the  shared-separation  concept.  The  ANOVA  indicated  that  there  were  no  statistically 
significant  main  effects  or  interactions  for  these  ratings. 

Section  G-9.  Frequency  of  URET  Trial  Plans 


The  researchers  performed  a  one-way  ANOVA  with  only  Control  Condition  (CO, 
CO:CDTI,  SS:L1,  SS:L2)  on  the  frequency  of  URET  trial  plans  (for  each  week  of  two  R- 
Side/D-Side  controller  teams).  The  ANOVA  indicated  that  the  effect  of  Control 
Condition  was  statistically  significant  [F(3, 6)  =  10.54,/?  <  .01].  The  differences  between 
the  four  Control  Condition  means  were  examined  using  Tukey  HSD  post-hoc 
comparisons.  The  results  indicated  that  significantly  more  trial  plans  were  formed  for 
CO  and  CO:CDTI  compared  to  SS:L2.  There  were  no  reliable  differences  between  CO, 
CO:CDTI,  and  SS:L1  and  no  differences  between  SS:L1  and  SS:L2. 

Section  G-10.  Controller  Ratings  for  Physical,  Mental,  and  Overall  Workload 

The  researchers  performed  two-way  ANOVAs  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for 
physical,  mental,  and  overall  workload.  The  ANOVAs  indicated  that  the  main  effect  of 
Control  Condition  was  statistically  significant  for  both  mental  workload  [F(3, 30)  =  6.59, 
p  <  .01],  and  overall  workload  [F(3, 30)  =  5.27,/?  <  .01].  No  other  main  effects  or 
interactions  were  significant  for  these  three  workload  ratings.  The  differences  between 
the  four  Control  Condition  means  for  both  mental  and  overall  workload  were  examined 
using  Tukey  HSD  post-hoc  comparisons.  The  results  indicated  that  mental  workload  was 
significantly  higher  for  SS:L1  compared  to  CO  and  SS:L2,  but  SS:L1  and  CO:CDTI  were 
not  statistically  different.  There  were  no  reliable  differences  between  CO,  CO:CDTI,  and 
SS:L2.  The  results  were  similar  for  overall  workload.  Overall  workload  was 
significantly  higher  for  SS:L1  compared  to  CO  and  SS:L2,  but  SS:L1  and  CO:CDTI  were 
not  statistically  different.  There  were  no  reliable  differences  between  CO,  CO:CDTI,  and 
SS:L2. 
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Section  G-1 1 .  Controller  Workload  Ratin2s  for  Maintainin2  Aircraft  Separation, 
rand  Line  Coordination.  R  Side-to-D  Side  Coordination.  Ground-to-Air 
Transmissions,  and  URET  Coordination 

The  researchers  performed  two-way  ANOVAs  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS;L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  workload  ratings  for 
maintaining  aircraft  separation,  radar-to-data  coordination,  and  URET  coordination.  The 
researchers  performed  one-way  ANOVAs  with  only  Control  Condition  (CO,  CO;CDTI, 
SS:L1,  SS:L2)  on  workload  ratings  for  ground-to-ground  coordination  (for  each  D-Side 
controller)  and  ground-to-air  communication  (for  each  R-Side  controller).  The  ANOVAs 
indicated  that  there  were  no  statistically  significant  main  effects  or  interactions  for  these 
five  workload  ratings. 

Section  G-12.  Controller  Ratines  for  Feeling  Rushed  and  Bored 

The  researchers  performed  two-way  ANOVAs  with  the  factors  of  Control  Condition 
(CO,  CO;CDTI,  SS;L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for 
feeling  rushed  and  bored  during  the  simulation.  The  ANOVAs  indicated  that  there  were 
no  statistically  significant  main  effects  or  interactions  for  these  two  ratings. 

Section  G-13.  Controller  Ratines  for  Overall  Situation  Awareness 

The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  controller  ratings  for 
their  level  of  overall  situation  awareness.  The  ANOVA  indicated  that  there  were  no 
statistically  significant  main  effects  or  interactions  for  these  ratings. 

Section  G-14.  Controller  Interval  Workload  Ratings 

The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  the  mean  workload 
ratings  across  the  intervals.  The  ANOVA  indicated  that  there  were  no  statistically 
significant  main  effects  or  interactions  for  these  ratings. 

Section  0-15.  Expert  Observer  Ratings  of  Controller  Physical  Taskload 

The  researchers  performed  a  two-way  ANOVA  with  the  factors  of  Control  Condition 
(CO,  CO:CDTI,  SS:L1,  SS:L2)  and  Position  (R-Side,  D-Side)  on  observer  ratings  of 
controller  physical  taskload.  The  ANOVA  indicated  that  the  main  effect  of  Control 
Condition  was  statistically  significant  [F(3, 30)  =  12.87, /;  <  .01].  No  other  mmn  effects 
or  interactions  were  significant  for  these  ratings.  The  differences  between  the  four 
Control  Condition  means  were  examined  using  Tukey  HSD  post-hoc  compmsons.  The 
results  indicated  that  observer  ratings  of  controller  physical  taskload  were  significantly 
higher  for  SS:L1  compared  to  CO,  CO:CDTI,  and  SS:L2.  There  were  no  reliable 
differences  between  CO,  CO:CDTI,  and  SS:L2. 
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Section  G-16A.  Controller  Ground-to-Air  and  Land  Line  PTTs 


The  researchers  performed  one-way  ANOVAs  with  only  Control  Condition  (CO, 
COrCDTI,  SS:L1,  SS:L2)  on  the  frequency  of  ground-to-air  transmissions  (for  each  R- 
Side  controller)  and  land  line  transmissions  (for  each  D-Side  controller).  TTie  ANOVAs 
indicated  that  the  effect  of  Control  Condition  was  statistically  significant  for  both  the 
frequency  of  ground-to-air  transmissions  [F(3, 9)  =  8.13,/?  <  .01],  and  land  line 
transmissions  [F(3, 9)  =  6.27,  p  <  .05].  TTie  differences  between  the  four  Control 
Condition  means  were  examined  using  Tukey  HSD  post-hoc  comparisons.  The  results 
indicated  that  the  frequency  of  ground-to-air  transmissions  was  significantly  lower  for 
SS;L2  compared  to  CO,  CO;CDTI,  and  SS:L1.  There  were  no  reliable  differences 
between  CO,  CO:CDTI,  and  SS:L1.  The  results  were  similar  for  land  line  transmissions. 
The  frequency  of  land  line  transmissions  was  significantly  lower  for  SS:L2  compared  to 
CO  and  SS:L1,  but  SS:L2  and  CO:CDTI  were  not  statistically  different.  There  were  no 
reliable  differences  between  CO,  CO:CDTI,  and  SS:L1. 

Section  G-16B,  Duration  per  Transmission  of  Ground-to-Air  and  Land  Line  PTTs 

The  researchers  performed  one-way  ANOVAs  with  only  Control  Condition  (CO, 
CO:CDTI,  SS:L1,  SS:L2)  on  the  duration  per  transmission  of  ground-to-air  transmissions 
(for  each  R-Side  controller)  and  land  line  transmissions  (for  each  D-Side  controller).  The 
ANOVAs  indicated  that  there  were  no  statistically  significant  main  effects  or  interactions 
for  these  two  ratings. 
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